在某些情况下，模型不仅依赖于说话内容，还依赖于指示其正在接受哪个基准测试的细微声学线索。因此，其得分夸大了模型在更广泛语音转录任务上的实际能力。为大规模检验这一现象，我们采用一组独立模型的集成方法，这些模型因其较低的音素错误率（PER）而被选出。PER 衡量书面转录文本与音频中声音的匹配程度，因而可作为模型忠实转录所听内容能力的有效代理指标。

In some cases, models appeared to rely not only on what was said, but also on subtle acoustic cues that indicated which benchmark they were being tested on. As a result, their scores overstated how well they could transcribe speech more generally. To test this at scale, we use an ensemble of independent models selected for their low phoneme error rate (PER). PER measures how closely a written transcription matches the sounds in the audio, making it a useful proxy for how faithfully a model transcribes what it hears.

在某些情况下，模型似乎不仅依赖于语音内容本身，还依赖于指示其正在接受哪个基准测试的细微声学线索。因此，其得分夸大了模型在更广泛场景下转录语音的真实能力。为大规模检验这一现象，我们采用了一个由多个独立模型组成的集成系统，这些模型均因其较低的音素错误率（PER）而被选中。PER 衡量书面转录文本与音频中实际发音的匹配程度，因而可作为模型忠实还原所听内容能力的有效代理指标。该集成系统的输出可用于标记那些所有模型均一致偏离基准参考文本的案例。随后，我们抽取部分此类标记案例，与人工标注结果进行比对，以验证经修正后的转录文本。当我们将相同内容以新采集的欧盟议会会议录音语音或通用语音重新呈现时，上述行为往往减弱甚至消失。在下方示例中，除一个模型外，其余所有模型在面对新议会录音克隆样本时，均恢复为忠实于音频的转录结果。这表明，模型正响应某些声学线索，从而识别出音频所属的基准测试集，并据此生成预期转录文本——即便该文本与实际音频内容相矛盾。Parakeet 是唯一一个在真实议会录音片段上复现基准文本、而在同说话人克隆样本上正确转录的模型。Phi-4 是唯一一个在 ep-fresh 克隆样本中仍遗漏敬语（courtesy）的模型。当我们改用与任何议会录音均无关联的通用 TTS 语音重新合成该句时，全部十一个模型均恢复了敬语。结果表明，该问题既普遍存在，又具有实质影响。我们的方法在所分析的 VoxPopuli 测试片段中，标记出 40% 存在潜在参考文本错误，影响约 3% 的全部参考词。 为进一步拓展基于共识分歧的探测方法，我们刻意在测试数据集的音频样本中静音数字，并要求模型转录其所“听到”的内容。由于数字在音频中实际缺失，模型不应输出任何数字，更不应准确复现原文中的数字。数字恢复率在公开基准测试中最高，在保留集或新采集音频（如下文 ep-fresh 和 libri-fresh）中则较低。在 LibriSpeech 上，部分在基准测试中表现最强的模型在约 30–40% 的样本中复现了已被静音的数字，尽管该数字本身已从音频中移除。该效应在若干模型的新采集数据上有所减弱，表明周边与基准测试相关联的声学环境（而不仅是文本层面的自动补全）有助于模型恢复参考文本。 在 LibriSpeech 内部，我们测试了一种涉及旧式空格规范的数据集内切换：部分参考文本写作 “any one”，另一些则写作 “anyone”。我们测量模型对给定变体的最低准确率，称之为“切换率”（switch rate）。若模型仅使用其中一种变体，则其切换率为 0%；若随机选择，则预期切换率为 50%；若模型能在每个测试样本中准确选用对应变体，则切换率为 100%。多个模型的切换率超过 50% 的随机基线，部分模型可达约 90% 的切换准确率。这表明，模型能够识别音频样本所属的数据集，并据此选用该基准测试所期望的拼写规范——尽管两种形式在听感上完全一致。 我们的研究发现表明，在两个主要开源数据集上，部分模型可检测与数据集关联的声学线索，并相应调整其转录行为。具体而言，模型可能复现音频中实际缺失但存在于参考文本中的词语，以异常高的比率恢复已被静音的数字，或利用周边声学上下文选择特定基准测试所预期的书面变体。 我们的研究发现亦提示，基准测试开发者应避免采用简单独立同分布（i.i.d.）的测试集划分方式，而应优先采用基于时间、说话人或其他元数据的分离策略。同时，提升训练数据与模型筛选流程的透明度，也将有助于研究人员理解此类行为的成因。 公开基准测试依然具有重要价值：它们透明、可复现、易于运行，且已被研究社区广泛理解。但其效用最大化，取决于我们能否有效区分真正提升语音转录能力的进步，与仅针对特定基准测试获得、却无法泛化至新音频的虚假增益。

In some cases, models appeared to rely not only on what was said, but also on subtle acoustic cues that indicated which benchmark they were being tested on. As a result, their scores overstated how well they could transcribe speech more generally. To test this at scale, we use an ensemble of independent models selected for their low phoneme error rate (PER). PER measures how closely a written transcription matches the sounds in the audio, making it a useful proxy for how faithfully a model transcribes what it hears. The ensemble results can be used to flag cases in which the models unanimously disagree with the benchmark's reference transcript. We then compare a sample of those flagged cases against human annotations to validate the corrected transcripts. When we present the same content in newly collected voices from EU parliamentary recordings or generic voices, this behavior often weakens or disappears. In the below samples, all but one model flips back to transcribing the audio-faithful transcript for a clone of a new parliamentary recording. This suggests that the models are responding to acoustic cues that help them identify the benchmark membership and thus produce the expected transcript even if it contradicts the audio. Parakeet is the only model that flips between reproducing the benchmark on the real clip and getting it right on the same-speaker clone. Phi-4 is the only model still dropping the courtesy on the ep-fresh clone. When we instead resynthesize the sentence in a generic TTS voice unconnected to any parliamentary recording, all eleven models restore the courtesy. The results suggest that this problem is both widespread and meaningful. Our methodology flagged potential reference errors in 40% of the VoxPopuli test clips we analyzed, affecting roughly 3% of all reference words. To build on the consensus disagreement probe, we deliberately silence numbers in the audio samples of test datasets and ask the models to transcribe what it hears. The number is literally absent from the audio, so models should not output any number, much less the exact number in the text. Recovery rates were highest on the public benchmarks and lower on held-out or newly collected audio (ep-fresh and libri-fresh below). On LibriSpeech, some of the strongest benchmark-performing models reproduced masked numbers in roughly 30–40% of examples, even though the number itself had been removed. The effect weakened on freshly collected data for several models, suggesting that the surrounding benchmark-associated audio—not only textual autocomplete—helped the models recover the reference. Within LibriSpeech, we test one intra-dataset switch involving an older spacing convention: some reference transcripts use "any one", while others use "anyone." We measure the minimum accuracy for a given variant, which we call "switch rate". If a model only uses one variant it would have a 0% switch rate; a model which picks randomly would be expected to have a 50% switch rate. A model which knows which variant to use in every test sample would earn a 100% switch rate. Multiple models exceed the 50% random-choice baseline, with some reaching roughly 90% switch accuracy. This suggests that the models can identify which dataset an audio sample comes from and select the spelling convention that benchmark expects, even though both forms sound identical. Our findings suggest that, on two major open-source datasets, some models detect dataset-associated acoustic cues and adjust their transcription behavior accordingly. Specifically, models may reproduce words that are absent from the audio but present in the reference transcript, recover silenced numbers at elevated rates, or use surrounding acoustic context to select the written variant expected by a particular benchmark. Our findings also suggest that benchmark developers should avoid simple independent and identically distributed test splits in favor of temporal, speaker, or other metadata-based separation. Greater transparency around training data and model-selection procedures would also help researchers understand how these behaviors arise. Public benchmarks remain valuable: they are transparent, repeatable, easy to run, and well understood by the research community. But they are most useful when we can distinguish genuine transcription improvements from benchmark-specific gains that do not generalize to new audio.