ORCA benchmark

Question Answering
Task Identifier: qa Cluster: Question Answering (QA) Data Type: MSA Score Metric: Macro F1-score Paper/GitHub/Website URL:

Rank	Submission Title	Model	URL	Score	Details
1	Huawei Noah Ark Lab MTL	JABERv2		63.24
2	Huawei Noah Ark Lab MTL	JABER		62.81
3	Huawei Noah Ark Lab Montreal	SABER		62.81
4	Huawei Noah Ark Lab MTL	AT5Bv2		62.81
5	Huawei Noah Ark Lab MTL	AT5B		61.95
6	Baselines	ARBERTv2		61.56
7	Baselines	AraBERTv02		61.5
8	Baselines	AraELECTRA		60.7
9	Baselines	GigaBERT-v4		58.71
10	Baselines	MARBERTv2		57.9
11	Baselines	GigaBERT-v3		57.74
12	Baselines	ARBERT		57.65
13	Baselines	AraBERTv02-Twitter		57.65
14	Baselines	CAMeLBERT-MSA		57.46
15	Baselines	AraBERTv01		56.79
16	Huawei Noah Ark Lab MTL	JABERv2-6L		56.57
17	Baselines	Arabic BERT		55.94
18	Huawei Noah Ark Lab MTL	AT5Sv2		54.7
19	Huawei Noah Ark Lab MTL	AT5S		54.66
20	Baselines	CAMeLBERT-MIX		53.98
21	Baselines	Arabic-ALBERT		53.89
22	Baselines	CAMeLBERT-CA		52.11
23	Baselines	MARBERT		49.35
24	Baselines	QARiB		45.27
25	Baselines	CAMeLBERT-DA		30.91

Submission details

Submission Name: Huawei Noah Ark Lab MTL
Model Name: JABERv2
GitHub/Model URL:

Model Description

Arabic BERT-base model pretrained on approximately 500GB of high-quality Arabic data. The source code and link to model weights will be updated on the GitHub page by (hopefully) the end of 2023 max.

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 63.24
ORCA score: 76.14
Submission Description: No parameter sharing, each task is finetuned independently. Also, there are no special tricks or additional data.

Submission details

Submission Name: Huawei Noah Ark Lab MTL
Model Name: JABER
GitHub/Model URL:

Model Description

JABER model in [this paper](https://aclanthology.org/2022.emnlp-main.205.pdf)

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 62.81
ORCA score: 75.48
Submission Description: No parameter sharing, each task is finetuned independently. Also, there are no special tricks or additional data.

Submission details

Submission Name: Huawei Noah Ark Lab Montreal
Model Name: SABER
GitHub/Model URL:

Model Description

SABER model in [this paper](https://aclanthology.org/2022.emnlp-main.205.pdf)

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 62.81
ORCA score: 77.09
Submission Description: No parameter sharing, each task is finetuned independently. Also, there are no special tricks or additional data.

Submission details

Submission Name: Huawei Noah Ark Lab MTL
Model Name: AT5Bv2
GitHub/Model URL:

Model Description

Arabic T5-base model pretrained on approximately 500GB of high-quality Arabic data. The source code and link to model weights will be updated on the GitHub page by (hopefully) the end of 2023 max.

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 62.81
ORCA score: 75.52
Submission Description: No parameter sharing, each task is finetuned independently. Also, there are no special tricks or additional data.

Submission details

Submission Name: Huawei Noah Ark Lab MTL
Model Name: AT5B
GitHub/Model URL:

Model Description

AT5B model described in [this paper](https://aclanthology.org/2022.emnlp-main.205.pdf)

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 61.95
ORCA score: 72.39
Submission Description: No parameter sharing, each task is finetuned independently. Also, there are no special tricks or additional data.

Submission details

Submission Name: Baselines
Model Name: ARBERTv2
GitHub/Model URL:

Model Description

ARBERTv2 is a BERT-based MLM with a vocabulary of 100K WordPieces, making ∼163M parameters. ARBERTv2 is the second release of ARBERT but with more data 243 GB of text (27.8B tokens).

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 61.56
ORCA score: 74.04
Submission Description: All finetuning parameters are described on ORCA paper

Submission details

Submission Name: Baselines
Model Name: AraBERTv02
GitHub/Model URL:

Model Description

The pretraining data used for the new AraBERT model is also used for Arabic GPT2 and ELECTRA. The dataset consists of 77GB or 200,095,961 lines or 8,655,948,860 words or 82,232,988,358 chars

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 61.5
ORCA score: 72.94
Submission Description: All finetuning parameters are described on ORCA paper

Submission details

Submission Name: Baselines
Model Name: AraELECTRA
GitHub/Model URL:

Model Description

The pretraining data used for the new AraELECTRA model is also used for AraGPT2 and AraBERTv2. The dataset consists of 77GB or 200,095,961 lines or 8,655,948,860 words or 82,232,988,358 chars

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 60.7
ORCA score: 71.17
Submission Description: All finetuning parameters are described on ORCA paper

Submission details

Submission Name: Baselines
Model Name: GigaBERT-v4
GitHub/Model URL:

Model Description

GigaBERT-v4 is a continued pre-training of GigaBERT-v3 on code-switched data, showing improved zero-shot transfer performance from English to Arabic on information extraction (IE) tasks.

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 58.71
ORCA score: 73.23
Submission Description: All finetuning parameters are described on ORCA paper

Submission details

Submission Name: Baselines
Model Name: MARBERTv2
GitHub/Model URL:

Model Description

MARBERT is a BERT-based MLM with a vocabulary of 100K WordPieces, making ∼163M parameters. MARBERT trained on a collection of Arabic Tweets comprising 29B tokens.

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 57.9
ORCA score: 70.89
Submission Description: All finetuning parameters are described on ORCA paper

Submission details

Submission Name: Baselines
Model Name: GigaBERT-v3
GitHub/Model URL:

Model Description

GigaBERT-v3 is a customized bilingual BERT for English and Arabic. It was pre-trained in a large-scale corpus (Gigaword+Oscar+Wikipedia) with ~10B tokens, showing state-of-the-art zero-shot transfer performance from English to Arabic on information extraction (IE) tasks.

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 57.74
ORCA score: 72.49
Submission Description: All finetuning parameters are described on ORCA paper

Submission details

Submission Name: Baselines
Model Name: ARBERT
GitHub/Model URL:

Model Description

ARBERT is a BERT-based MLM with a vocabulary of 100K WordPieces, making ∼163M parameters. ARBERT trained on a collection of Arabic datasets comprising 61GB of text (6.2B tokens).

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 57.65
ORCA score: 73.45
Submission Description: All finetuning parameters are described on ORCA paper

Submission details

Submission Name: Baselines
Model Name: AraBERTv02-Twitter
GitHub/Model URL:

Model Description

AraBERTv0.2-Twitter-base/large are two new models for Arabic dialects and tweets, trained by continuing the pre-training using the MLM task on ~60M Arabic tweets (filtered from a collection on 100M).

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 57.65
ORCA score: 72.56
Submission Description: All finetuning parameters are described on ORCA paper

Submission details

Submission Name: Baselines
Model Name: CAMeLBERT-MSA
GitHub/Model URL:

Model Description

CAMeLBERT is a collection of BERT models pre-trained on Arabic texts with different sizes and variants. We release pre-trained language models for Modern Standard Arabic (MSA), dialectal Arabic (DA), and classical Arabic (CA), in addition to a model pre-trained on a mix of the three

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 57.46
ORCA score: 73.35
Submission Description: All finetuning parameters are described on ORCA paper

Submission details

Submission Name: Baselines
Model Name: AraBERTv01
GitHub/Model URL:

Model Description

The pretraining data used for the new AraBERT model is also used for Arabic GPT2 and ELECTRA. The dataset consists of 77GB or 200,095,961 lines or 8,655,948,860 words or 82,232,988,358 chars

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 56.79
ORCA score: 71.46
Submission Description: All finetuning parameters are described on ORCA paper

Submission details

Submission Name: Huawei Noah Ark Lab MTL
Model Name: JABERv2-6L
GitHub/Model URL:

Model Description

Arabic BERT 6 layer model pretrained on approximately 500GB of high-quality Arabic data. The source code and link to model weights will be updated on the GitHub page by (hopefully) the end of 2023 max.

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 56.57
ORCA score: 73.72
Submission Description: No parameter sharing, each task is finetuned independently. Also, there are no special tricks or additional data.

Submission details

Submission Name: Baselines
Model Name: Arabic BERT
GitHub/Model URL:

Model Description

arabic-bert-base model was pretrained on ~8.2 Billion words

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 55.94
ORCA score: 71.9
Submission Description: All finetuning parameters are described on ORCA paper

Submission details

Submission Name: Huawei Noah Ark Lab MTL
Model Name: AT5Sv2
GitHub/Model URL:

Model Description

Arabic T5-small model pretrained on approximately 500GB of high-quality Arabic data. The source code and link to model weights will be updated on the GitHub page by (hopefully) the end of 2023 max.

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 54.7
ORCA score: 72.68
Submission Description: No parameter sharing, each task is finetuned independently. Also, there are no special tricks or additional data.

Submission details

Submission Name: Huawei Noah Ark Lab MTL
Model Name: AT5S
GitHub/Model URL:

Model Description

AT5S model described in [this paper](https://aclanthology.org/2022.emnlp-main.205.pdf)

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 54.66
ORCA score: 71.02
Submission Description: No parameter sharing, each task is finetuned independently. Also, there are no special tricks or additional data.

Submission details

Submission Name: Baselines
Model Name: CAMeLBERT-MIX
GitHub/Model URL:

Model Description

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 53.98
ORCA score: 71.13
Submission Description: All finetuning parameters are described on ORCA paper

Submission details

Submission Name: Baselines
Model Name: Arabic-ALBERT
GitHub/Model URL:

Model Description

The model is pretrained on ~4.4 Billion words

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 53.89
ORCA score: 58.14
Submission Description: All finetuning parameters are described on ORCA paper

Submission details

Submission Name: Baselines
Model Name: CAMeLBERT-CA
GitHub/Model URL:

Model Description

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 52.11
ORCA score: 68.25
Submission Description: All finetuning parameters are described on ORCA paper

Submission details

Submission Name: Baselines
Model Name: MARBERT
GitHub/Model URL:

Model Description

MARBERT is a BERT-based MLM with a vocabulary of 100K WordPieces, making ∼163M parameters. MARBERT trained on a collection of Arabic Tweets comprising 128 GB of text (15.6 B tokens).

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 49.35
ORCA score: 69.91
Submission Description: All finetuning parameters are described on ORCA paper

Submission details

Submission Name: Baselines
Model Name: QARiB
GitHub/Model URL:

Model Description

QCRI Arabic and Dialectal BERT (QARiB) model, was trained on a collection of ~ 420 Million tweets and ~ 180 Million sentences of text.

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 45.27
ORCA score: 69.43
Submission Description: ARBERT

Submission details

Submission Name: Baselines
Model Name: CAMeLBERT-DA
GitHub/Model URL:

Model Description

Submission Details

Number of Submitted Tasks: 29
Task (Question Answering)Score: 30.91
ORCA score: 66.1
Submission Description: All finetuning parameters are described on ORCA paper