今天我们推出 BenchMIRT，一种在单个提示（即模型被评分的问题与任务）层面审计大语言模型（LLM）基准测试的新方法。BenchMIRT 帮助研究人员分离各类信号，从而识别出真正驱动基准测试得分的因素。该方法通过分析模型在每个问题或任务上的表现，并估计与正确作答最密切相关的底层能力来实现这一目标。

Today we’re introducing BenchMIRT, a new method for auditing LLM benchmarks at the level of individual prompts—the questions and tasks a model is scored on. BenchMIRT helps researchers separate those signals and see what’s actually driving a benchmark’s score. It does this by analyzing how models perform on each question or task and estimating which underlying capabilities are most closely associated with getting it right.

今天，我们推出 BenchMIRT——一种在单个提示（即模型被评分所依据的问题与任务）层面审计大语言模型（LLM）基准的新方法。BenchMIRT 帮助研究人员分离不同信号，从而看清究竟是哪些因素真正驱动了基准的得分。其原理在于分析模型在每个问题或任务上的表现，并估计哪些潜在能力与答对问题最密切相关。BenchMIRT 借鉴了项目反应理论（Item Response Theory, IRT），该方法起源于心理测量学——即通过测试作答模式来测量能力与特质的学科。IRT 基于一个简单理念：并非每道题目都能同等程度地反映应试者的能力。有些题目更难，有些则更能有效区分能力强弱不同的应试者。此前，研究人员已将单维 IRT 应用于单个基准，包括我们在 Fluid Benchmarking 工作中的实践。BenchMIRT 将该方法拓展至多维项目反应理论（Multidimensional IRT, MIRT），从而可分离出可能共同影响同一问题作答表现的多种能力。 BenchMIRT 在模型与问题两个层面同时应用 IRT。对于给定模型，它估计该模型在所选基准所涵盖各项能力上的强弱程度；对于每道题目，它估计该题的难度，以及其在区分在相关能力上更强或更弱的模型方面的效度。关键在于，我们并未向 BenchMIRT 明确告知哪些基准对应哪些能力。它独立识别出两个主导维度：安全性（safety）与通用推理（general reasoning）。当我们从头重复整个分析时，这同一组两个维度始终稳定浮现，表明该结果具有稳健性，而非偶然依赖于某次特定分析。 对于许多基准，BenchMIRT 的分析基本印证了其预设目标：在推理类基准中表现优异者，确实展现出更强的推理能力；而在越狱（jailbreak）与有害内容（harmful-content）类基准中表现优异者，则确实体现出更高的安全性。BBQ 基准用于评估社会偏见，常被归入安全性基准，但在 BenchMIRT 分析中却与通用推理维度高度对齐。这意味着较低的 BBQ 得分，可能部分源于受试模型在理解或推理某些题目方面存在困难，而不仅反映其安全性行为本身。 这些发现并不必然意味着现有基准存在缺陷或不完整；相反，它们表明单一基准得分可能融合了若干不同信号，而 BenchMIRT 可帮助解耦这些信号，使得分更易于解读。 Harmbench 中各题目的项目难度与区分度在两个维度上的分布。维度 0 表征安全性维度，维度 1 对应通用推理维度。 BenchMIRT 还可协助识别评估中哪些题目最能反映基准旨在测量的能力。借助 BenchMIRT 提供的题目级估计值，我们对用于训练 BenchMIRT 的相同 16 个基准中的全部题目进行排序，并筛选出在区分强弱模型方面表现最优的题目，同时仍兼顾难易题目的合理混合。在这些基准中，仅保留 10% 的题目，通常即可近乎完整复现全量题目集所呈现的模型在底层安全性或推理能力上的强弱排序；而保留 50% 的题目时，其对上述能力的衡量结果往往比全量题目集更为贴近。 BenchMIRT 还可利用其在模型与题目间学习到的模式，预测某模型在未观测过的基准题目上的作答表现。在我们的实验中，它对留出题目的正确作答预测准确率达 79%。相比之下，一种更简单的基线方法——假设模型在每道题目上的表现与其在该基准整体上的平均表现一致——准确率仅为 70%。实践中，这意味着 BenchMIRT 能够基于已掌握的模型能力特征及各题目的认知要求，更精准地估算模型表现，而无需让每个模型回答所有题目。 BenchMIRT 为研究人员更深入理解并优化其用以评估模型能力的基准提供了一种新路径。通过聚焦单个题目而非仅关注总体得分，它可揭示基准是否混杂了不同能力、识别出行为异于其余题目的题目簇，并凸显那些对基准本应测量之能力贡献甚微的题目。 当然，该方法也存在权衡取舍。若目标仅为按模型在随机留出题目上的预测表现进行排序，则基准的平均得分略优于 BenchMIRT。BenchMIRT 的优势在于其对单个题目表现所提供的更精细刻画。但这种题目级细节亦具双面性：同一套估计值既可用于识别基准中最富信息量的安全性题目，也可能被用于剔除这些题目，从而生成一个更易被不安全模型通过的薄弱评估。现有工具已具备类似裁剪评估的能力；我们认为，提升对基准题目实际测量内容的透明度，其价值足以抵消这一风险——尽管该风险真实存在。 尽管如此，我们仍将 BenchMIRT 及其后续同类工具视为迈向更具针对性的基准设计与更高效评估的重要一步。通过揭示究竟哪些题目真正驱动了基准结果，此类方法有望助力研究人员构建规模更小、焦点更集中、解释性更强的评估体系，同时更清晰地呈现其所意在测量的能力本质。

Today we’re introducing BenchMIRT, a new method for auditing LLM benchmarks at the level of individual prompts—the questions and tasks a model is scored on. BenchMIRT helps researchers separate those signals and see what’s actually driving a benchmark’s score. It does this by analyzing how models perform on each question or task and estimating which underlying capabilities are most closely associated with getting it right. BenchMIRT takes cues from Item Response Theory (IRT), a technique originating in psychometrics—the field concerned with measuring abilities and traits from patterns of test responses. IRT starts from a simple idea: not every question tells you the same amount about the person taking a test. Some are harder than others, and some do a better job of distinguishing stronger performers from weaker ones. Researchers have previously applied single-dimensional IRT to individual benchmarks, including in our Fluid Benchmarking work. BenchMIRT extends that approach with multidimensional IRT, or MIRT, allowing it to separate multiple capabilities that may contribute to performance on the same questions. BenchMIRT applies IRT at both the model and question level. For a given model, it estimates the model’s strength on the capabilities reflected across the selected benchmarks. For each question, it estimates how difficult the question is and how well it distinguishes models that are stronger or weaker on those capabilities. Crucially, we didn’t tell BenchMIRT which benchmarks were measuring which capabilities. It independently recovered two dominant dimensions: safety and general reasoning. When we repeated our analysis from scratch, those same two dimensions emerged each time, suggesting the result was stable rather than specific to one analysis. For many benchmarks, BenchMIRT largely confirmed their intended focus: strong performance on reasoning benchmarks tracked with reasoning ability, while strong performance on jailbreak and harmful-content benchmarks tracked with safety. BBQ, which evaluates social bias and is commonly grouped with safety benchmarks, aligned much more strongly with general reasoning in BenchMIRT’s analysis. That means a low BBQ score may partly reflect difficulty understanding or reasoning through certain questions, rather than safety behavior alone. These findings don’t necessarily mean the benchmarks are flawed or incomplete. Rather, they show that a single benchmark score can combine several different signals—and that BenchMIRT can help disentangle those signals and make the score easier to interpret. Item difficulty and discrimination in both dimensions for Harmbench. Dimension 0 models the safety dimension, while Dimension 1 maps to the general reasoning dimension. BenchMIRT can also help identify which questions in an evaluation are most informative about the capability the benchmark is trying to measure. Using BenchMIRT’s question-level estimates, we ranked questions across the same 16 benchmarks used to train BenchMIRT and kept those that did the best job of distinguishing stronger from weaker models, while still preserving a mix of easier and harder questions. Across those benchmarks, keeping only 10% of the questions generally preserved nearly the same picture of which models were stronger or weaker on the underlying safety or reasoning capability as using the full set. Keeping 50% of the questions often matched the full benchmark’s measure of those capabilities even more closely. BenchMIRT can also use the patterns it learns across models and questions to predict how a model would perform on a benchmark question it hasn’t been observed answering. In our experiments, it correctly predicted whether a model would answer a held-out question correctly 79% of the time. By comparison, a simpler approach that assumes a model will perform on each question about as well as it does on the benchmark overall was correct 70% of the time. In practice, that means BenchMIRT can estimate model performance more precisely from what it has already learned about the model’s abilities and the demands of each question, without needing to evaluate every model on every question. BenchMIRT offers a way to better understand and refine the benchmarks researchers use to evaluate model capabilities. By looking at individual questions rather than only overall scores, it can reveal when a benchmark mixes together different capabilities, identify clusters of questions that behave differently from the rest, and surface questions that add little useful information about the capability the benchmark is meant to measure. There are trade-offs, too. If the goal is to rank models by their predicted performance on randomly held-out items, the benchmark’s average score performs slightly better than BenchMIRT. BenchMIRT’s advantage is the finer-grained picture it provides of performance on individual questions. That question-level detail can also cut both ways: the same estimates that help identify a benchmark’s most informative safety questions could be used to remove them, producing a weaker evaluation that an unsafe model could pass. Existing tools already make it possible to trim evaluations in similar ways, and we think the added transparency into what benchmark questions are actually measuring is worth that risk—but it’s a real one. Still, we see BenchMIRT – and future tools like it – as a step toward more targeted benchmark design and efficient evaluation. By showing which questions are actually driving a benchmark’s results, these approaches could help researchers build evaluations that are smaller, more focused, and easier to interpret, while giving a clearer picture of the capabilities they’re meant to measure.