Submission Name: Huawei Noah Ark Lab MTL
Model Name: AT5Bv2
GitHub/Model URL:    

---

Model Description

---

Arabic T5-base model pretrained on approximately 500GB of high-quality Arabic data. The source code and link to model weights will be updated on the GitHub page by (hopefully) the end of 2023 max.

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 71.97
ORCA score: 75.52
Submission Description: No parameter sharing, each task is finetuned independently. Also, there are no special tricks or additional data.

Submission Name: Huawei Noah Ark Lab MTL
Model Name: AT5B
GitHub/Model URL:    

---

Model Description

---

AT5B model described in [this paper](https://aclanthology.org/2022.emnlp-main.205.pdf)

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 71.66
ORCA score: 72.39
Submission Description: No parameter sharing, each task is finetuned independently. Also, there are no special tricks or additional data.

Submission Name: Huawei Noah Ark Lab MTL
Model Name: JABERv2
GitHub/Model URL:    

---

Model Description

---

Arabic BERT-base model pretrained on approximately 500GB of high-quality Arabic data. The source code and link to model weights will be updated on the GitHub page by (hopefully) the end of 2023 max.

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 71.37
ORCA score: 76.14
Submission Description: No parameter sharing, each task is finetuned independently. Also, there are no special tricks or additional data.

Submission Name: Huawei Noah Ark Lab MTL
Model Name: JABER
GitHub/Model URL:    

---

Model Description

---

JABER model in [this paper](https://aclanthology.org/2022.emnlp-main.205.pdf)

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 71.27
ORCA score: 75.48
Submission Description: No parameter sharing, each task is finetuned independently. Also, there are no special tricks or additional data.

Submission Name: Huawei Noah Ark Lab MTL
Model Name: SABER
GitHub/Model URL:    

---

Model Description

---

Arabic BERT-large model pretrained on approximately 500GB of high-quality Arabic data. The source code and link to model weights will be updated on the GitHub page by (hopefully) the end of 2023 max.

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 71.06
ORCA score: 77.09
Submission Description: No parameter sharing, each task is finetuned independently. Also, there are no special tricks or additional data.

Submission Name: Baselines
Model Name: AraELECTRA
GitHub/Model URL:    

---

Model Description

---

The pretraining data used for the new AraELECTRA model is also used for AraGPT2 and AraBERTv2. The dataset consists of 77GB or 200,095,961 lines or 8,655,948,860 words or 82,232,988,358 chars

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 70.2
ORCA score: 71.17
Submission Description: All finetuning parameters are described on ORCA paper

Submission Name: Baselines
Model Name: ARBERTv2
GitHub/Model URL:    

---

Model Description

---

ARBERTv2 is a BERT-based MLM with a vocabulary of 100K WordPieces, making ∼163M parameters. ARBERTv2 is the second release of ARBERT but with more data 243 GB of text (27.8B tokens).

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 68.17
ORCA score: 74.04
Submission Description: All finetuning parameters are described on ORCA paper

Submission Name: Baselines
Model Name: AraBERTv02
GitHub/Model URL:    

---

Model Description

---

The pretraining data used for the new AraBERT model is also used for Arabic GPT2 and ELECTRA. The dataset consists of 77GB or 200,095,961 lines or 8,655,948,860 words or 82,232,988,358 chars

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 67.93
ORCA score: 72.94
Submission Description: All finetuning parameters are described on ORCA paper

Submission Name: Baselines
Model Name: ARBERT
GitHub/Model URL:    

---

Model Description

---

ARBERT is a BERT-based MLM with a vocabulary of 100K WordPieces, making ∼163M parameters. ARBERT trained on a collection of Arabic datasets comprising 61GB of text (6.2B tokens).

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 67.22
ORCA score: 73.45
Submission Description: All finetuning parameters are described on ORCA paper

Submission Name: Baselines
Model Name: GigaBERT-v3
GitHub/Model URL:    

---

Model Description

---

GigaBERT-v3 is a customized bilingual BERT for English and Arabic. It was pre-trained in a large-scale corpus (Gigaword+Oscar+Wikipedia) with ~10B tokens, showing state-of-the-art zero-shot transfer performance from English to Arabic on information extraction (IE) tasks.

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 67.14
ORCA score: 72.49
Submission Description: All finetuning parameters are described on ORCA paper

Submission Name: Huawei Noah Ark Lab MTL
Model Name: JABERv2-6L
GitHub/Model URL:    

---

Model Description

---

Arabic BERT 6 layer model pretrained on approximately 500GB of high-quality Arabic data. The source code and link to model weights will be updated on the GitHub page by (hopefully) the end of 2023 max.

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 67.1
ORCA score: 73.72
Submission Description: No parameter sharing, each task is finetuned independently. Also, there are no special tricks or additional data.

Submission Name: Baselines
Model Name: AraBERTv02-Twitter
GitHub/Model URL:    

---

Model Description

---

AraBERTv0.2-Twitter-base/large are two new models for Arabic dialects and tweets, trained by continuing the pre-training using the MLM task on ~60M Arabic tweets (filtered from a collection on 100M).

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 66.67
ORCA score: 72.56
Submission Description: All finetuning parameters are described on ORCA paper

Submission Name: Baselines
Model Name: GigaBERT-v4
GitHub/Model URL:    

---

Model Description

---

GigaBERT-v4 is a continued pre-training of GigaBERT-v3 on code-switched data, showing improved zero-shot transfer performance from English to Arabic on information extraction (IE) tasks.

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 66.53
ORCA score: 73.23
Submission Description: All finetuning parameters are described on ORCA paper

Submission Name: Baselines
Model Name: CAMeLBERT-MSA
GitHub/Model URL:    

---

Model Description

---

CAMeLBERT is a collection of BERT models pre-trained on Arabic texts with different sizes and variants. We release pre-trained language models for Modern Standard Arabic (MSA), dialectal Arabic (DA), and classical Arabic (CA), in addition to a model pre-trained on a mix of the three

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 66.02
ORCA score: 73.35
Submission Description: All finetuning parameters are described on ORCA paper

Submission Name: Huawei Noah Ark Lab MTL
Model Name: AT5S
GitHub/Model URL:    

---

Model Description

---

AT5S model described in [this paper](https://aclanthology.org/2022.emnlp-main.205.pdf)

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 65.72
ORCA score: 71.02
Submission Description: No parameter sharing, each task is finetuned independently. Also, there are no special tricks or additional data.

Submission Name: Huawei Noah Ark Lab MTL
Model Name: AT5Sv2
GitHub/Model URL:    

---

Model Description

---

Arabic T5-small model pretrained on approximately 500GB of high-quality Arabic data. The source code and link to model weights will be updated on the GitHub page by (hopefully) the end of 2023 max.

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 64.84
ORCA score: 72.68
Submission Description: No parameter sharing, each task is finetuned independently. Also, there are no special tricks or additional data.

Submission Name: Baselines
Model Name: AraBERTv01
GitHub/Model URL:    

---

Model Description

---

The pretraining data used for the new AraBERT model is also used for Arabic GPT2 and ELECTRA. The dataset consists of 77GB or 200,095,961 lines or 8,655,948,860 words or 82,232,988,358 chars

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 64.69
ORCA score: 71.46
Submission Description: All finetuning parameters are described on ORCA paper

Submission Name: Baselines
Model Name: CAMeLBERT-MIX
GitHub/Model URL:    

---

Model Description

---

CAMeLBERT is a collection of BERT models pre-trained on Arabic texts with different sizes and variants. We release pre-trained language models for Modern Standard Arabic (MSA), dialectal Arabic (DA), and classical Arabic (CA), in addition to a model pre-trained on a mix of the three

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 63.82
ORCA score: 71.13
Submission Description: All finetuning parameters are described on ORCA paper

Submission Name: Baselines
Model Name: MARBERTv2
GitHub/Model URL:    

---

Model Description

---

MARBERT is a BERT-based MLM with a vocabulary of 100K WordPieces, making ∼163M parameters. MARBERT trained on a collection of Arabic Tweets comprising 29B tokens.

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 62.22
ORCA score: 70.89
Submission Description: All finetuning parameters are described on ORCA paper

Submission Name: Baselines
Model Name: Arabic BERT
GitHub/Model URL:    

---

Model Description

---

arabic-bert-base model was pretrained on ~8.2 Billion words

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 62.15
ORCA score: 71.9
Submission Description: All finetuning parameters are described on ORCA paper

Submission Name: Baselines
Model Name: Arabic-ALBERT
GitHub/Model URL:    

---

Model Description

---

The model is pretrained on ~4.4 Billion words

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 61.62
ORCA score: 58.14
Submission Description: All finetuning parameters are described on ORCA paper

Submission Name: Baselines
Model Name: CAMeLBERT-CA
GitHub/Model URL:    

---

Model Description

---

CAMeLBERT is a collection of BERT models pre-trained on Arabic texts with different sizes and variants. We release pre-trained language models for Modern Standard Arabic (MSA), dialectal Arabic (DA), and classical Arabic (CA), in addition to a model pre-trained on a mix of the three

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 61.18
ORCA score: 68.25
Submission Description: All finetuning parameters are described on ORCA paper

Submission Name: Baselines
Model Name: MARBERT
GitHub/Model URL:    

---

Model Description

---

MARBERT is a BERT-based MLM with a vocabulary of 100K WordPieces, making ∼163M parameters. MARBERT trained on a collection of Arabic Tweets comprising 128 GB of text (15.6 B tokens).

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 60.5
ORCA score: 69.91
Submission Description: All finetuning parameters are described on ORCA paper

Submission Name: Baselines
Model Name: QARiB
GitHub/Model URL:    

---

Model Description

---

QCRI Arabic and Dialectal BERT (QARiB) model, was trained on a collection of ~ 420 Million tweets and ~ 180 Million sentences of text.

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 55.67
ORCA score: 69.43
Submission Description: ARBERT

Submission Name: Baselines
Model Name: CAMeLBERT-DA
GitHub/Model URL:    

---

Model Description

---

CAMeLBERT is a collection of BERT models pre-trained on Arabic texts with different sizes and variants. We release pre-trained language models for Modern Standard Arabic (MSA), dialectal Arabic (DA), and classical Arabic (CA), in addition to a model pre-trained on a mix of the three

---

Submission Details

---

Number of Submitted Tasks: 29
Task (XLNI)Score: 54.29
ORCA score: 66.1
Submission Description: All finetuning parameters are described on ORCA paper