Measured 21 November 2026 on the full FLEURS test
Hebrew speech-to-text accuracy benchmark
On all 792 clips of the Hebrew FLEURS test, measured on 21 November 2026, KolWrite Oracle Max had a word error rate of 12.6% (95% interval 11.2% to 14.1%), the lowest of the 22 engines in the comparison; the next lowest published figure is 17.4%. Every transcript is published, so you can check the score.
Word error rate of every engine in the comparison
12.6% word error rate, 95% interval 11.2% to 14.1%KolWrite Oracle Max on all 792 clips of the Hebrew FLEURS test, 21 November 2026
Oracle Max made 1,734 word errors (967 substitutions, 510 deletions, 257 insertions) over 13,751 reference words in 792 clips, 2.05 hours of speech. Resampling the 792 clips 10,000 times gives a 95% interval of 11.2% to 14.1%. 309 of the 792 sentences were transcribed with no error.
For 16 of the 21 engines the leaderboard publishes the transcript of every clip, and one further file for a timing run of an ivrit.ai model. We scored those 17 files and ours with the same code, then resampled the 792 clips 10,000 times, using the same resampled clips for both sides of each comparison. Oracle Max had the lower error rate than every one of the 17 in all 10,000 resamples. The other 5 engines publish no per-clip file for this test set, so they can be compared only by their published figures; all 5 are above the top of Oracle Max's interval, 14.1%.
All results
| Engine, as the leaderboard names it | Word error rate | 95% interval | Sentences with no error | Lead, points (95% interval) | Fewer word errors |
|---|---|---|---|---|---|
| KolWrite Oracle Max | 12.6% | 11.2% to 14.1% | 309 | – | – |
| ivrit.ai whisper-large-v3-ct2-20250513 | 17.4% | published figure | – | – | – |
| Soniox (stt) | 17.7% | 16.2% to 19.3% | 192 | 5.1 (4.3 to 5.9) | 29% |
| ElevenLabs Scribe v1 | 18.1% | 16.6% to 19.7% | 218 | 5.5 (4.7 to 6.4) | 31% |
| ivrit.ai whisper-large-v3-turbo-ct2-20250513 | 18.1% | published figure | – | – | – |
| ivrit.ai whisper-large-v3-turbo (timing run) | 18.3% | 16.8% to 19.8% | 166 | 5.7 (5.0 to 6.4) | 31% |
| ivrit.ai whisper-large-v3-ct2-20250403 | 20.0% | published figure | – | – | – |
| ivrit.ai whisper-large-v3-ct2-20250209 | 20.8% | published figure | – | – | – |
| ivrit.ai whisper-large-v3-turbo-ct2-20250403 | 20.9% | 19.4% to 22.5% | 117 | 8.3 (7.5 to 9.2) | 40% |
| OpenAI gpt-4o-transcribe | 21.1% | 19.5% to 22.8% | 166 | 8.5 (7.6 to 9.4) | 40% |
| Amazon Transcribe (batch) | 23.0% | 21.4% to 24.6% | 101 | 10.4 (9.5 to 11.3) | 45% |
| Deepgram Nova-3 | 23.4% | 21.8% to 24.9% | 105 | 10.7 (9.8 to 11.7) | 46% |
| zipformer2-66m-streaming-he | 23.4% | 21.9% to 25.1% | 108 | 10.8 (9.9 to 11.7) | 46% |
| ivrit.ai faster-whisper-v2-d4 | 24.2% | 22.7% to 25.8% | 93 | 11.6 (10.7 to 12.6) | 48% |
| ivrit.ai whisper-large-v3-turbo-ct2-20250209 | 24.5% | published figure | – | – | – |
| ivrit.ai faster-whisper-v2-d3-e3 | 25.6% | 24.0% to 27.2% | 76 | 13.0 (12.0 to 14.0) | 51% |
| Whisper large-v3 (faster-whisper) | 26.2% | 24.6% to 27.8% | 65 | 13.6 (12.6 to 14.6) | 52% |
| Whisper large-v2 (faster-whisper) | 26.7% | 25.1% to 28.3% | 73 | 14.0 (13.1 to 15.1) | 53% |
| Amazon Transcribe (streaming) | 28.7% | 27.0% to 30.4% | 62 | 16.1 (15.0 to 17.1) | 56% |
| Whisper large-v3-turbo (faster-whisper) | 28.9% | 27.2% to 30.6% | 65 | 16.3 (15.1 to 17.5) | 56% |
| OpenAI gpt-4o-mini-transcribe | 30.1% | 28.3% to 31.9% | 82 | 17.5 (16.2 to 18.8) | 58% |
| Google Speech | 38.5% | 36.8% to 40.4% | 32 | 25.9 (24.6 to 27.3) | 67% |
| nemotron-3.5-asr-streaming-0.6b | 46.0% | 43.7% to 48.4% | 32 | 33.4 (31.2 to 35.6) | 73% |
Our scoring of the engines' per-clip files differs from the leaderboard's published figures by at most 0.14 percentage points on this set. The row marked "timing run" is a further file of an ivrit.ai model, scored from the leaderboard's timing results; it is not a 22nd engine and is left out of the chart.
How it was measured
- Product. KolWrite cloud transcription, tier Oracle Max, called through the public KolWrite API on the production service on 21 November 2026, with the language set to Hebrew, speaker separation off and no custom vocabulary. These are the calls a customer makes.
- Test data. The Hebrew (he_il) test split of FLEURS (Google, CC BY 4.0), dataset revision 70bb2e84b976, all 792 clips. Nothing was left out. One clip returned no text; it is scored as all errors.
- Scoring. The method of the ivrit.ai Hebrew transcription leaderboard (normaliser
HebrewTextNormalizerin ivrit-ai/asr-training, commite333b4a0): one error rate over all the words of all clips, after removing vowel points, punctuation, quote marks and letter case. Before the run, our scoring code reproduced the figures in the leaderboard's table to within 0.17 percentage points. - Statistics. Percentile bootstrap over clips, 10,000 resamples, random seed 20261121 (numpy
default_rng). In each comparison, both engines are scored on the same resampled clips. - The other engines. The leaderboard's own runs, in the versions it names, as published on 14 October 2026 (the leaderboard, commit
412599da3e). We did not run them. - Publisher. ITANU Studios Ltd, the maker of KolWrite. We measured our own product, and we publish the files so that anyone can check the score.
What this does not show
- One test set: read sentences, one speaker per clip. Recordings of meetings, phone calls, noise and overlapping speakers are different, and the order of the engines can differ on them.
- One tier, one language and one run. A transcription service does not return identical text on every run, so a repeat will land near 12.6%, inside or close to the interval, and not exactly on it.
- FLEURS is a public test set, for Oracle Max as for every engine in the table.
- The leaderboard also reports four other Hebrew sets. Their licences or access terms do not let us publish a run on them, so none is shown here.
- The other engines' figures are those of the versions and dates the leaderboard lists. Newer versions were not measured. Speed and price are not measured here.
- Only Hebrew is shown. Other languages appear when they are measured and their per-clip file is public; this page says nothing about how Oracle Max ranks in a language that is not listed.
Check it yourself
1. Download the per-clip file. oracle-max-hebrew-fleurs-792-2026-11-21.csv has all 792 rows: the FLEURS reference sentence, what Oracle Max returned, both texts after cleaning, and the error counts of each clip (id is the clip's row number in the FLEURS test split). SHA-256: df61915b0e8604e070f6619983ddcd945cd20a6bd004ec5f13e8fd04a5e66900. The reference sentences are from FLEURS (Conneau et al., 2022), © Google, used under CC BY 4.0; our transcripts in the file may be reused under the same licence.
2. Recompute the score. score_wer.py (SHA-256 619048fbded7676e1e2b7f84e3771b89ea9a8affa05674713e94f135ef4ed1c9) reads that file with Python's standard library, recounts every clip's errors from the cleaned texts and prints the figures:
python3 score_wer.py oracle-max-hebrew-fleurs-792-2026-11-21.csv
Prints
clips: 792
reference words: 13751
word errors: 1734
word error rate: 12.6%
95% interval: 11.2% to 14.1% (10000 resamples over clips, seed 20261121)
sentences with no error: 309
recount of errors from the cleaned texts: 792 of 792 clips agree with the file
The interval is drawn with Python's random generator and the published seed, so it can differ from the published interval in the last digit.
3. Run Oracle Max on your own account. Create an API key and add credits, then send the clips of the FLEURS Hebrew test split one by one (about US$7.38 for all 792 at US$3.60 per audio hour). With the API of this site (model oracle_max, language he, no speaker labels, no custom vocabulary):
# 1. Upload one clip (WAV, under 4 MiB). The answer contains fileUrl.
curl -X POST https://app.kolwrite.com/upload \
-H "x-api-key: $KOLWRITE_API_KEY" \
-H "Content-Type: application/json" \
-d "{\"filename\": \"clip.wav\", \"contentType\": \"audio/wav\", \"content\": \"$(base64 < clip.wav | tr -d '\n')\"}"
# 2. Start the job with that fileUrl. The answer contains jobId.
curl -X POST https://app.kolwrite.com/transcribe \
-H "x-api-key: $KOLWRITE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"audio_url": "FILE_URL_FROM_STEP_1", "model": "oracle_max", "language": "he", "output_format": "json", "word_timestamps": false}'
# 3. Poll every 5 to 10 seconds until status is COMPLETED. Join result.segments[].text with spaces.
curl https://app.kolwrite.com/transcribe/JOB_ID -H "x-api-key: $KOLWRITE_API_KEY"
Our measurement went through the KolWrite platform API (https://api-prod.kolwrite.com, authenticated with an access key from the app's API access settings, documented here), where the same tier is called oracle-max. This is the exact request it sent for each clip:
curl -X POST https://api-prod.kolwrite.com/api/v1/uploads \
-H "Authorization: Bearer $KOLWRITE_KEY" -F "[email protected];type=audio/wav"
# the answer's data.upload_id goes into the job:
curl -X POST https://api-prod.kolwrite.com/api/v1/jobs \
-H "Authorization: Bearer $KOLWRITE_KEY" -H "Content-Type: application/json" \
-d '{"type": "transcription", "input": {"type": "upload_ref", "upload_id": "UPLOAD_ID"}, "profile": "oracle-max", "options": {"language": "he", "speaker_separation": {"mode": "none"}}}'
# poll GET /api/v1/jobs/JOB_ID until data.status is "completed"; the text is data.result.segments[].text
Where the data lives. Dataset: google/fleurs, revision 70bb2e84b976. Leaderboard and its per-clip files: ivrit-ai/hebrew-transcription-leaderboard, commit 412599da3e. Scoring code: ivrit-ai/asr-training, commit e333b4a0. Our own note on the result: oracle-max-hebrew-fleurs-792-2026-11-21.md.
Engine and product names belong to their owners and are used only to report the results the leaderboard published for them.
Questions and answers
Is this benchmark independent?
No. ITANU Studios Ltd, the maker of KolWrite, ran it on its own product. The other engines' figures are the leaderboard's, not ours. We publish every transcript and the scoring method so that anyone can check the Oracle Max score, and the leaderboard's own files let anyone check the rest.
Why is only Hebrew listed?
A language is listed only when we can publish a per-clip file for it that anyone can rescore. More languages are being measured, and we promise no date.
Will I get exactly the same figure if I run it myself?
Close to it, not exactly. A transcription service does not return identical text on every run, so a repeat should land near 12.6%, inside or near the interval.
Where do I report a mistake on this page?
Write to [email protected] with the clip id or the figure. We correct the page and the file and say what changed.