AI Monitoring in Healthcare: Why Voice Intelligence Matters More Than Ever
Modulate ML Team Announces New Public Entity Transcription Benchmark
September 10, 2026
Parker Ennis
(HE/HIM/HIS)
Most speech benchmarks don't measure whether a system gets names right. Ours does, it's free to use, and you can read below to see how we score on it.
Context is king
A transcript that gets 98% of its words right can still be useless. If the 2% it misses are “Worcester,” (sounds nothing like it’s spelled) “Nguyen,” (a common name that’s notoriously mis-transcribed) and “Chime,” (a company name a recognizer will happily write as a lowercase common word) you have a redaction pipeline that leaks, a search index that can't surface the conversation, and a CRM record filed against the wrong account. Nobody notices the words a system gets right. They notice the name and the things it gets wrong.
More often than not, speech recognition gets graded on word error rate (WER). WER counts the fraction of words a system gets wrong, and it treats every word the same. Missing “the” and missing “Baghdady” score identically. That's a reasonable way to measure transcription in general and a poor way to measure the part most products actually run on.
If you're building on transcription, those aren't the same mistake. The transcript is rarely the finished product, it's an input to redaction, search, routing, CRM enrichment, and moderation. Those systems key off proper nouns, so a wrong name doesn't produce a slightly worse transcript, it produces a clean-looking transcript and a wrong result downstream. Proper nouns are a small share of any transcript. A model can improve its WER while getting worse at exactly the words those systems run on, and standard evaluation won't tell you what happened. So our ML team built something that does.
Our ML team recently published a dataset that measures per-entity transcription accuracy and is free for anyone to download and use, along with our own results and the categories where we come second.
Why names are the hardest words in a transcript
Getting names right is a well-known hard problem in speech recognition, and the reason is structural. Speech models use surrounding context to resolve ambiguous audio. Given a phrase like “I'll see you on ___day,” the model knows a weekday goes there, so it’s choosing between seven options for each day of the week. Proper nouns, though, don't work that way. Surnames, uncommon places and organization names are where this breaks down, and they're disproportionately the words an audio pipeline needs.
The industry already knows this, which is why almost every transcription API sells keyword boosting. You can hand the system your expected names up front and it does better on them. That feature is an admission that base models miss names, and it only helps when you already know which names are coming. Most real workloads don't.
Announcing the Entity Transcription Benchmark
Modulate built this measurement as a public, independently testable way to see how you stack up. The Entity Transcription Benchmark is live on Hugging Face today and is free to download or use. It holds 2,151 clips, six hours of audio, and 3,245 annotated entity spans across two deliberately different kinds of speech:
- Spontaneous multi-speaker recordings from US municipal and legislative meetings. Real rooms, real crosstalk, verbatim references with the disfluencies left in.
- Volunteers reading encyclopedic prose. Clean audio, clean references.
The metric is per-entity accuracy: out of the names actually spoken in the audio, what fraction does a system get right? Higher is better in this case. It's a different question from WER that gives different answers.
Every entity carries a type (people, organizations, countries and cities, other physical places, groups, and other named things like events, works, laws, vehicles, and landmarks) and a difficulty tier. Tier A is the discriminative set: multi-token names, rare surnames, uncommon organizations and places. Tier B is the trivial set, mostly bare first names like David and Jim, or high-frequency demonyms and countries like British, German, and France . Any English recognizer handles tier B, so including it inflates a score without separating one system from another. Tier A is what we report. You can re-tier or report both tiers yourself.
You can download the dataset, run it against any system you like, publish the numbers you get, and build on it commercially. Attribution and share-alike are the only conditions. There is no gate to use it.
How we built it, and why you can check our work
The audio isn't ours and that’s by design. It's redistributed unmodified from two openly licensed research corpora: MLCommons People's Speech and FLEURS, following Belebele’s passage selection. Those are datasets other groups built and released for public research, and they're widely used across the field. We didn't record any of it, didn't “game the system” to suit our own model, and can't quietly change it later because the clips come from public sources. Anyone can pull the same clips from the original sources and confirm they match ours.
What we added on top is the annotation layer to describe which words in each clip are names, what kind of name each one is, how hard it is to get right, and a scorer that decides whether a transcript got it or not.
The dataset card is also blunt about how the annotations were made. The spans and types were generated automatically rather than by a human annotation team, and validated by a single reviewer who is also an author of the dataset. We ship the working files too, including the review sheet, every type decision, and the duplicate readings we removed. If you think an annotation is wrong, open the file, read the judgement, and tell us. A disputed annotation is more useful to us than silent acceptance.
What it shows: our English model against Deepgram nova-3
We built this to check our own work so we ran ourselves through it first. Deepgram is the alternative most teams weigh us against, so we ran them too, on the same audio and with the same scorer.
We ran our English batch endpoint, velma-2-stt-batch-english-vfast, against Deepgram nova-3, their flagship model, with punctuation and capitalization enabled. Both systems ran once using their default settings.
In the spirit of transparency, we don’t win in every area. There are several places where Modulate has room to grow and improve. On named events, works and laws in meeting audio, Deepgram beats us 70.5% to 57.4% across 61 spans. Separately, one operational result worth as much as any accuracy figure is that nova-3 returned an empty transcript on 34 clips, whereas we returned empty on 5.
Also, one caveat we'd rather state than have pointed out: we didn't turn on Deepgram's keyword boosting, the feature described earlier, and switching it on would likely raise their scores. It's a per-customer tuning step rather than a default and so leaving it off keeps the comparison like for like. However, you should know the option is there.
How we scored each file
For every name spoken in a clip, we ask whether the transcript contains that name spelled correctly. Nothing else in the sentence counts. Get every other word wrong and the named entity right, and you score a hit. Names also have to match in full. "Newt Gingrick" is not a hit for "Newt Gingrich."
A system shouldn't lose points for formatting, so we normalize both transcripts identically before scoring. Capitalization and punctuation are ignored, numbers are written out one consistent way so "six one seven" and "617" count as the same answer, and common abbreviations are expanded. Applying it to both sides is what keeps it neutral. It's also why absolute scores move if you score with your own rules.
Two rules that keep the numbers honest:
- Clips where both systems returned nothing are left out
- We don't report a category in a subset unless it has enough examples there to mean anything
And neither set of transcripts was edited after collection.
Why our headline number is 77.9% and not 92.7%
Our lead on the public read-speech corpora is 12.9 points. On the independent meeting audio it's 2.1. That difference is why the dataset has two subsets, and it's an important aspect to take away from this.
All three sources are public, so training exposure can't be ruled out for any model, ours included. The meeting audio sits outside the standard benchmark suites, which makes it the harder and more honest test. A big lead on public data that shrinks to almost nothing on harder, spontaneous audio is exactly the pattern this benchmark exists to expose.
That being said, the headline number is 77.9% and not 92.7%. A benchmark that only flattered us would be worth nothing to you (and not much to us either at the end of the day). If you're evaluating transcription vendors, that's the one question worth carrying into every conversation: was this number measured on audio the model could already have seen?
Run it and see for yourself
The self-test confirms the matcher is behaving before you trust any number it gives you. Then pass in your own transcripts as a CSV and it scores them the same way it scored ours.
Use the shipped scorer rather than writing your own. Entity accuracy is very sensitive to normalization, casing, punctuation and spoken numbers included, so an independent scorer won't give you comparable numbers.
Run it against us, against Deepgram, against Google or AssemblyAI, against your own model. You should get the same answers we did. If you don't, we want to hear about it.
Try it out
At the end of the day, names, proper nouns, and the likes are the part of a transcript your users notice when it breaks, but now there's a public way to check who gets them right. We encourage you to give this new dataset to benchmark a try and validate the results for yourself!
Let us know what you think and send us any feedback at support@modulate.ai.
And if you haven't already, consider signing up for the Modulate API with up to 400 free hours of transcription.
