Measured 21 November 2026 on the full FLEURS test, next to the figures the AURA leaderboard publishes
Arabic speech-to-text accuracy benchmark
On all 428 clips of the Arabic FLEURS (ar_eg) test, measured on 21 November 2026, KolWrite Oracle Max had a word error rate of 4.25% (95% interval 3.29% to 5.43%), below each of the 24 figures the leaderboard publishes for this test; the lowest of them is cohere-transcribe-arabic-07-2026 at 4.78%, which lies inside Oracle Max's interval. The leaderboard publishes totals only, so no ranking is claimed. Every transcript is published, so you can check the score.
Oracle Max next to the leaderboard's published figures
4.25% word error rate under the AURA text rules, 95% interval 3.29% to 5.43%KolWrite Oracle Max on all 428 clips of the Arabic FLEURS (ar_eg) test, 21 November 2026
Oracle Max made 340 word errors (227 substitutions, 75 deletions, 38 insertions) over 8,002 reference words in 428 clips, 1.3 hours of speech. Resampling the 428 clips 10,000 times gives a 95% interval of 3.29% to 5.43%. 287 of the 428 clips were transcribed with no error.
Show all 26 enginesShow fewer
The leaderboard publishes totals only, so its 24 figures are published figures only: there is no per-clip file, no interval and no paired comparison for them. Set against Oracle Max's interval of 3.29% to 5.43%, 1 inside it and 23 above it. Because their transcripts are not public, a figure inside our interval is not shown to be better or worse than ours. One engine was run by us on the same clips with the same text rules, whisper-large-v3 (run by us). We scored its transcripts and ours with the same code and resampled the 428 clips 10,000 times, using the same resampled clips on both sides. Oracle Max had the lower error rate in all 10,000 resamples, by 5.70 points (95% interval 4.80 to 6.61).
Where this puts Oracle Max. Its 4.25% is below each of the 24 figures the leaderboard publishes for this test. The lowest of them, cohere-transcribe-arabic-07-2026 at 4.78%, lies inside Oracle Max's interval, so against that figure the difference is not shown to be clear. The leaderboard publishes totals only, so this page claims no ranking.
All results
| Engine, as the leaderboard names it | Word error rate | 95% interval | Clips with no error | Oracle Max lead, points (95% interval) | Against Oracle Max |
|---|---|---|---|---|---|
| KolWrite Oracle Max | 4.25% | 3.29% to 5.43% | 287 | – | – |
| cohere-transcribe-arabic-07-2026 | 4.78% | published only | – | – | Figure inside our interval |
| aura_stt_stream_xl_deployment_WS_API_onprem1s | 5.51% | published only | – | – | Figure above our interval |
| aura_stt_stream_xl_beam64 | 5.53% | published only | – | – | Figure above our interval |
| aura_stt_stream_xl_deployment_WS_API_onprem0.5s | 5.60% | published only | – | – | Figure above our interval |
| aura_stt_stream_L_beam4 | 5.64% | published only | – | – | Figure above our interval |
| aura_stt_offline_xl_deployment_offline_API_onprem_30s | 5.98% | published only | – | – | Figure above our interval |
| aura_stt_offline_xl_deployment_offline_API_gcp_60s | 5.98% | published only | – | – | Figure above our interval |
| aura_stt_offline_xl | 5.99% | published only | – | – | Figure above our interval |
| aura_stt_stream_L_greedy | 5.99% | published only | – | – | Figure above our interval |
| jais_asr | 7.09% | published only | – | – | Figure above our interval |
| aura_stt_stream_m_greedy | 7.60% | published only | – | – | Figure above our interval |
| aura_stt_stream_m_beam4 | 7.80% | published only | – | – | Figure above our interval |
| aura_stt_v1 | 8.79% | published only | – | – | Figure above our interval |
| aura_stt_lf1 | 8.87% | published only | – | – | Figure above our interval |
| nvidia_conformer_ctc_arabic_lm | 9.00% | published only | – | – | Figure above our interval |
| whisper3 | 9.33% | published only | – | – | Figure above our interval |
| whisper-large-v3 (run by us) | 9.95% | 8.82% to 11.23% | 124 | 5.70 (4.80 to 6.61) | Lower for Oracle Max |
| qwen3_asr_1_7b | 10.60% | published only | – | – | Figure above our interval |
| nvidia_nemotron_streaming_06b (langtag specified, 1.12s) | 11.22% | published only | – | – | Figure above our interval |
| nvidia_nemotron_streaming_06b (langtag specified, 320ms) | 11.47% | published only | – | – | Figure above our interval |
| nvidia_nemotron_streaming_06b (320ms) | 11.64% | published only | – | – | Figure above our interval |
| gemma4_4b_it | 16.21% | published only | – | – | Figure above our interval |
| qwen2_5_omni_7b | 17.62% | published only | – | – | Figure above our interval |
| qwen3_asr_0_6b | 18.68% | published only | – | – | Figure above our interval |
| qwen2_5_omni_3b | 27.40% | published only | – | – | Figure above our interval |
Our run of whisper-large-v3 scores 9.95%, where the leaderboard publishes 9.33% for its own run of that model: our scoring is 0.6 points stricter on that engine, not looser.
How it was measured
- Product. KolWrite cloud transcription, tier Oracle Max, called through the public KolWrite API on the production service on 21 November 2026, with the language set to Arabic, speaker separation off and no custom vocabulary. These are the calls a customer makes.
- Test data. The Egyptian Arabic (ar_eg) test split of FLEURS (Google, CC BY 4.0), dataset revision 70bb2e84b976, all 428 clips, 1.3 hours of read sentences. The reference is the dataset's raw transcription column. Nothing was left out. All 428 clips returned text.
- Scoring. The text rules that the AURA leaderboard states for its Arabic scores, written by us as code because the leaderboard publishes none (the code is on this page, below): Arabic diacritics removed; Arabic and Latin letters, digits, @ and % kept; a full stop or comma kept only between two digits; an apostrophe kept only between two characters; everything else turned into a space; the alef variants written as a bare alef; Latin letters in lower case; spaces collapsed. We also remove the tatweel, the stretching mark. Then one error rate over all the words of all clips, by plain word edit distance. The cleaned texts of every clip are in the per-clip file.
- Statistics. Percentile bootstrap over clips, 10,000 resamples, random seed 20261121 (NumPy
default_rng). In each paired comparison, both engines are scored on the same resampled clips. - The other engines. The 24 figures of the QCRI AURA Arabic-English leaderboard for this test, as published on 22 October 2026 (commit
22d708a7), under the names it gives them. The leaderboard publishes totals only, so these are published figures only: we did not run those engines and cannot pair them. The one engine we ran ourselves is whisper-large-v3 (run by us), on the same 428 clips with the same text rules, compared with Oracle Max over 10,000 paired resamples. As a check of our scoring it scores 9.95% where the leaderboard publishes 9.33% for its own run of that model, so our scoring is 0.6 points stricter on that engine, not looser. - Publisher. ITANU Studios Ltd, the maker of KolWrite. We measured our own product, and we publish the files so that anyone can check the score.
What this does not show
- One test set of read or prepared speech, one speaker per clip. Recordings of meetings, phone calls, noise and overlapping speakers are different, and the order of the engines can differ on them.
- One tier, one language and one run. A transcription service does not return identical text on every run, so a repeat will land near 4.25%, inside or close to the interval, and not exactly on it. The interval covers the sample of clips, not the run-to-run variation.
- The leaderboard publishes one total per engine and no per-clip outputs. Oracle Max's figure is below each of the 24, but the lowest of them, cohere-transcribe-arabic-07-2026 at 4.78%, lies inside our interval. This page does not rank Oracle Max against those engines.
- The figure depends on the text cleaning. With the cleaning of the Open ASR Leaderboard's basic multilingual mode, which keeps letter variants such as the alef forms apart, the same 428 transcripts score 9.70% (95% interval 8.39% to 11.15%). Whichever cleaning you use, state it with the figure. We publish 4.25% because the AURA leaderboard defines its Arabic scores with the rules above.
- FLEURS is read sentences in one configuration of Arabic. It says nothing about dialects, conversation or noisy recordings, and FLEURS has been public since 2022, so any large model may have met its sentences in training.
- Our code of the AURA rules is our reading of a description. It is published in full so that you can read it, but it is not the leaderboard's own code.
- The other engines' figures are those of the versions and dates the leaderboard lists. Newer versions were not measured. Speed and price are not measured here.
- This page says nothing about how Oracle Max ranks in a language or on a test that is not listed on the benchmark overview.
Check it yourself
1. Download the per-clip file. oracle-max-arabic-fleurs-428-2026-11-21.csv has all 428 rows: the reference sentence, what Oracle Max returned, both texts after cleaning, and the error counts of each clip (id is the clip's identifier in the test set). SHA-256: 93b76d34cfb20d3e895b0d30f5b6677b6993d2086c650217fa69ba6a6d39fbab. The reference sentences are from FLEURS (Conneau et al., 2022), © Google, used under CC BY 4.0; our transcripts in the file may be reused under the same licence.
2. Recompute the score. score_wer.py (SHA-256 070513ff9c648f3ba8c9f44aa9950a0f82a8228ff6740722c81004a8257a0cf3) reads that file with Python's standard library, recounts every clip's errors from the cleaned texts and prints the figures:
python3 score_wer.py oracle-max-arabic-fleurs-428-2026-11-21.csv
Prints
clips: 428
reference words: 8002
word errors: 340
word error rate: 4.25%
95% interval: 3.29% to 5.43% (10000 resamples over clips, seed 20261121)
clips with no error: 287
recount of errors and words from the cleaned texts: 428 of 428 clips agree with the file
The interval is the published one: the script redraws the same 10,000 resamples with NumPy's random generator (seed 20261121), reimplemented in plain Python, so it needs no package. The recount is a plain word edit distance.
The text cleaning, in full. This function turns the reference and the transcript of every clip into the norm_reference_text and norm_predicted_text of the file; run on the first two text columns it reproduces the last two for all 428 clips.
import re
import unicodedata
DIACRITICS = re.compile("[\u0610-\u061A\u064B-\u065F\u0670\u06D6-\u06ED]")
def normalise(text):
text = DIACRITICS.sub("", text)
text = text.replace("\u0640", "") # tatweel
text = re.sub("[\u0622\u0623\u0625\u0671\u0672\u0673]", "\u0627", text) # alef variants -> bare alef
out = []
for i, ch in enumerate(text):
before = text[i - 1] if i else ""
after = text[i + 1] if i + 1 < len(text) else ""
if ch in ".," and before.isdigit() and after.isdigit():
out.append(ch)
elif ch in "'\u2019" and before and after and not before.isspace() and not after.isspace():
out.append("'")
elif ch in "@%" or ch.isspace() or ch.isdigit():
out.append(ch)
elif "\u0600" <= ch <= "\u06FF" and unicodedata.category(ch).startswith("L"):
out.append(ch)
elif ch.isascii() and ch.isalpha():
out.append(ch.lower())
else:
out.append(" ")
return re.sub(r"\s+", " ", "".join(out)).strip()
3. Run Oracle Max on your own account. Create an API key and add credits, then send the clips of the FLEURS Arabic (ar_eg) test split one by one (about US$4.68 for all 428 at US$3.60 per audio hour). With the API of this site (model oracle_max, language ar, no speaker labels, no custom vocabulary):
# 1. Upload one clip (WAV, under 4 MiB). The answer contains fileUrl.
curl -X POST https://app.kolwrite.com/upload \
-H "x-api-key: $KOLWRITE_API_KEY" \
-H "Content-Type: application/json" \
-d "{\"filename\": \"clip.wav\", \"contentType\": \"audio/wav\", \"content\": \"$(base64 < clip.wav | tr -d '\n')\"}"
# 2. Start the job with that fileUrl. The answer contains jobId.
curl -X POST https://app.kolwrite.com/transcribe \
-H "x-api-key: $KOLWRITE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"audio_url": "FILE_URL_FROM_STEP_1", "model": "oracle_max", "language": "ar", "output_format": "json", "word_timestamps": false}'
# 3. Poll every 5 to 10 seconds until status is COMPLETED. Join result.segments[].text with spaces.
curl https://app.kolwrite.com/transcribe/JOB_ID -H "x-api-key: $KOLWRITE_API_KEY"
Our measurement went through the KolWrite platform API (https://api-prod.kolwrite.com, authenticated with an access key from the app's API access settings, documented here), where the same tier is called oracle-max. This is the exact request it sent for each clip:
curl -X POST https://api-prod.kolwrite.com/api/v1/uploads \
-H "Authorization: Bearer $KOLWRITE_KEY" -F "[email protected];type=audio/wav"
# the answer's data.upload_id goes into the job:
curl -X POST https://api-prod.kolwrite.com/api/v1/jobs \
-H "Authorization: Bearer $KOLWRITE_KEY" -H "Content-Type: application/json" \
-d '{"type": "transcription", "input": {"type": "upload_ref", "upload_id": "UPLOAD_ID"}, "profile": "oracle-max", "options": {"language": "ar", "speaker_separation": {"mode": "none"}}}'
# poll GET /api/v1/jobs/JOB_ID until data.status is "completed"; the text is data.result.segments[].text
Where the data lives. Dataset: FLEURS, revision 70bb2e84b976. Leaderboard and its per-clip files: QCRI/AURA-Arabic-English-ASR-leaderboard, commit 22d708a7.
Engine and product names belong to their owners and are used only to report the results the leaderboard published for them.
Questions and answers
Is this benchmark independent?
No. ITANU Studios Ltd, the maker of KolWrite, ran it on its own product. The other engines' figures are the leaderboard's, not ours, apart from the one engine we ran ourselves. We publish every transcript and the scoring method so that anyone can check the Oracle Max score.
Which languages does this site list?
The languages in which Oracle Max has the lowest word error rate on a named public test: Hebrew, Spanish, French, Arabic, German and Portuguese. Where it is level with the lowest, or below the totals a leaderboard publishes, the page says so. Other languages are not listed.
Will I get exactly the same figure if I run it myself?
Close to it, not exactly. A transcription service does not return identical text on every run, so a repeat should land near 4.25%, inside or near the interval.
Where do I report a mistake on this page?
Write to [email protected] with the clip id or the figure. We correct the page and the file and say what changed.
How does Oracle Max compare in Arabic?
On this test its figure, 4.25%, is below every figure the AURA leaderboard publishes. The lowest of them, cohere-transcribe-arabic-07-2026 at 4.78%, lies inside the interval of Oracle Max. The leaderboard publishes totals only, so we claim no ranking, and the figure depends on the text cleaning this page states.