How to build a better AI benchmark

It’s not casual being 1 of Silicon Valley’s favourite benchmarks. 

SWE-Bench (pronounced “swee bench”) launched successful November 2024 to measure an AI model’s coding skill, utilizing much than 2,000 real-world programming problems pulled from the nationalist GitHub repositories of 12 antithetic Python-based projects. 

In the months since then, it’s rapidly go 1 of the astir fashionable tests successful AI. A SWE-Bench people has go a mainstay of large exemplary releases from OpenAI, Anthropic, and Google—and extracurricular of instauration models, the fine-tuners astatine AI firms are successful changeless contention to spot who tin emergence supra the pack. The apical of the leaderboard is simply a pileup betwixt 3 antithetic good tunings of Anthropic’s Claude Sonnet exemplary and Amazon’s Q developer agent. Auto Code Rover—one of the Claude modifications—nabbed the fig 2 spot successful November, and was acquired conscionable 3 months later.

Despite each the fervor, this isn’t precisely a truthful appraisal of which exemplary is “better.” As the benchmark has gained prominence, “you commencement to spot that radical truly privation that apical spot,” says John Yang, a researcher connected the squad that developed SWE-Bench astatine Princeton University. As a result, entrants person begun to crippled the system—which is pushing galore others to wonderment whether there’s a amended mode to really measurement AI achievement.

Developers of these coding agents aren’t needfully doing thing arsenic straightforward cheating, but they’re crafting approaches that are excessively neatly tailored to the specifics of the benchmark. The archetypal SWE-Bench trial acceptable was constricted to programs written successful Python, which meant developers could summation an vantage by grooming their models exclusively connected Python code. Soon, Yang noticed that high-scoring models would neglect wholly erstwhile tested connected antithetic programming languages—revealing an attack to the trial that helium describes arsenic “gilded.”

“It looks bully and shiny astatine archetypal glance, but past you effort to tally it connected a antithetic connection and the full happening conscionable benignant of falls apart,” Yang says. “At that point, you’re not designing a bundle engineering agent. You’re designing to marque a SWE-Bench agent, which is overmuch little interesting.”

The SWE-Bench contented is simply a grounds of a much sweeping—and complicated—problem successful AI evaluation, and 1 that’s progressively sparking heated debate: The benchmarks the manufacture uses to usher improvement are drifting further and further distant from evaluating existent capabilities, calling their basal worth into question. Making the concern worse, respective benchmarks, astir notably FrontierMath and Chatbot Arena, person precocious travel nether vigor for an alleged deficiency of transparency. Nevertheless, benchmarks inactive play a cardinal relation successful exemplary development, adjacent if fewer experts are consenting to instrumentality their results astatine look value. OpenAI cofounder Andrej Karpathy recently described the concern arsenic “an valuation crisis”: the manufacture has less trusted methods for measuring capabilities and nary wide way to amended ones. 

“Historically, benchmarks were the mode we evaluated AI systems,” says Vanessa Parli, manager of probe astatine Stanford University’s Institute for Human-Centered AI. “Is that the mode we privation to measure systems going forward? And if it's not, what is the way?”

A increasing radical of academics and AI researchers are making the lawsuit that the reply is to spell smaller, trading sweeping ambition for an attack inspired by the societal sciences. Specifically, they privation to absorption much connected investigating validity, which for quantitative societal scientists refers to however good a fixed questionnaire measures what it’s claiming to measure—and, much fundamentally, whether what it is measuring has a coherent definition. That could origin occupation for benchmarks assessing hazily defined concepts similar “reasoning” oregon “scientific knowledge”—and for developers aiming to scope the much-hyped extremity of artificial wide intelligence—but it would enactment the manufacture connected firmer crushed arsenic it looks to beryllium the worthy of idiosyncratic models.

“Taking validity earnestly means asking folks successful academia, industry, oregon wherever to amusement that their strategy does what they accidental it does,” says Abigail Jacobs, a University of Michigan prof who is simply a cardinal fig successful the caller propulsion for validity. “I deliberation it points to a weakness successful the AI satellite if they privation to backmost disconnected from showing that they tin enactment their claim.”

The limits of accepted testing

If AI companies person been dilatory to respond to the increasing nonaccomplishment of benchmarks, it’s partially due to the fact that the test-scoring attack has been truthful effectual for truthful long. 

One of the biggest aboriginal successes of modern AI was the ImageNet challenge, a benignant of antecedent to modern benchmarks. Released successful 2010 arsenic an unfastened situation to researchers, the database held much than 3 cardinal images for AI systems to categorize into 1,000 antithetic classes.

Crucially, the trial was wholly agnostic to methods, and immoderate palmy algorithm rapidly gained credibility careless of however it worked. When an algorithm called AlexNet broke done successful 2012, with a past unconventional signifier of GPU training, it became 1 of the foundational results of modern AI. Few would person guessed successful beforehand that AlexNet’s convolutional neural nets would beryllium the concealed to unlocking representation recognition—but aft it scored well, nary 1 dared quality it. (One of AlexNet’s developers, Ilya Sutskever, would spell connected to cofound OpenAI.)

A ample portion of what made this situation truthful effectual was that determination was small applicable quality betwixt ImageNet’s entity classification situation and the existent process of asking a machine to admit an image. Even if determination were disputes astir methods, nary 1 doubted that the highest-scoring exemplary would person an vantage erstwhile deployed successful an existent representation designation system.

But successful the 12 years since, AI researchers person applied that aforesaid method-agnostic attack to progressively wide tasks. SWE-Bench is commonly utilized arsenic a proxy for broader coding ability, portion different exam-style benchmarks often basal successful for reasoning ability. That wide scope makes it hard to beryllium rigorous astir what a circumstantial benchmark measures—which, successful turn, makes it hard to usage the findings responsibly. 

Where things interruption down

Anka Reuel, a PhD pupil who has been focusing connected the benchmark occupation arsenic portion of her probe astatine Stanford, has go convinced the valuation occupation is the effect of this propulsion toward generality. “We’ve moved from task-specific models to general-purpose models,” Reuel says. “It’s not astir a azygous task anymore but a full clump of tasks, truthful valuation becomes harder.”

Like the University of Michigan’s Jacobs, Reuel thinks “the main contented with benchmarks is validity, adjacent much than the applicable implementation,” noting: “That’s wherever a batch of things interruption down.” For a task arsenic analyzable arsenic coding, for instance, it’s astir intolerable to incorporated each imaginable script into your occupation set. As a result, it’s hard to gauge whether a exemplary is scoring amended due to the fact that it’s much skilled astatine coding oregon due to the fact that it has much efficaciously manipulated the occupation set. And with truthful overmuch unit connected developers to execute grounds scores, shortcuts are hard to resist.

For developers, the anticipation is that occurrence connected tons of circumstantial benchmarks volition adhd up to a mostly susceptible model. But the techniques of agentic AI mean a azygous AI strategy tin encompass a analyzable array of antithetic models, making it hard to measure whether betterment connected a circumstantial task volition pb to generalization. “There’s conscionable galore much knobs you tin turn,” says Sayash Kapoor, a machine idiosyncratic astatine Princeton and a salient professional of sloppy practices successful the AI industry. “When it comes to agents, they person benignant of fixed up connected the champion practices for evaluation.”

In a paper from past July, Kapoor called retired circumstantial issues successful however AI models were approaching the WebArena benchmark, designed by Carnegie Mellon University researchers successful 2024 arsenic a trial of an AI agent’s quality to traverse the web. The benchmark consists of much than 800 tasks to beryllium performed connected a acceptable of cloned websites mimicking Reddit, Wikipedia, and others. Kapoor and his squad identified an evident hack successful the winning model, called STeP. STeP included circumstantial instructions astir however Reddit structures URLs, allowing STeP models to leap straight to a fixed user’s illustration leafage (a predominant constituent of WebArena tasks).

This shortcut wasn’t precisely cheating, but Kapoor sees it arsenic “a superior misrepresentation of however good the cause would enactment had it seen the tasks successful WebArena for the archetypal time.” Because the method was successful, though, a akin argumentation has since been adopted by OpenAI’s web cause Operator. (“Our valuation mounting is designed to measure however good an cause tin lick tasks fixed immoderate acquisition astir website structures and task execution,” an OpenAI typical said erstwhile reached for comment. “This attack is accordant with however others person utilized and reported results with WebArena.” STeP did not respond to a petition for comment.)

Further highlighting the occupation with AI benchmarks, precocious past period Kapoor and a squad of researchers wrote a insubstantial that revealed important problems successful Chatbot Arena, the fashionable crowdsourced valuation system. According to the paper, the leaderboard was being manipulated; galore apical instauration models were conducting undisclosed backstage investigating and releasing their scores selectively.

Today, adjacent ImageNet itself, the parent of each benchmarks, has started to autumn unfortunate to validity problems. A 2023 study from researchers astatine the University of Washington and Google Research recovered that erstwhile ImageNet-winning algorithms were pitted against six real-world information sets, the architecture betterment “resulted successful small to nary progress,” suggesting that the outer validity of the trial had reached its limit.

Going smaller

For those who judge the main occupation is validity, the champion hole is reconnecting benchmarks to circumstantial tasks. As Reuel puts it, AI developers “have to edifice to these high-level benchmarks that are astir meaningless for downstream consumers, due to the fact that the benchmark developers can’t expect the downstream task anymore.” So what if determination was a mode to assistance the downstream consumers place this gap?

In November 2024, Reuel launched a nationalist ranking task called BetterBench, which rates benchmarks connected dozens of antithetic criteria, specified arsenic whether the codification has been publically documented. But validity is simply a cardinal theme, with peculiar criteria challenging designers to spell retired what capableness their benchmark is investigating and however it relates to the tasks that marque up the benchmark.

“You request to person a structural breakdown of the capabilities,” Reuel says. “What are the existent skills you attraction about, and however bash you operationalize them into thing we tin measure?”

The results are surprising. One of the highest-scoring benchmarks is besides the oldest: the Arcade Learning Environment (ALE), established successful 2013 arsenic a mode to trial models’ quality to larn however to play a room of Atari 2600 games. One of the lowest-scoring is the Massive Multitask Language Understanding (MMLU) benchmark, a wide utilized trial for wide connection skills; by the standards of BetterBench, the transportation betwixt the questions and the underlying accomplishment was excessively poorly defined.

Irene Solaiman, Hugging Face’s caput of planetary policy, says the radical volition absorption connected gathering valid benchmarks that spell beyond measuring straightforward capabilities. “There’s conscionable truthful overmuch hunger for a bully benchmark disconnected the support that already works,” Solaiman says. “A batch of evaluations are trying to bash excessively much.”

Increasingly, the remainder of the manufacture seems to agree. In a paper successful March, researchers from Google, Microsoft, Anthropic, and others laid retired a caller model for improving evaluations—with validity arsenic the archetypal step. 

“AI valuation subject must,” the researchers argue, “move beyond coarse grained claims of ‘general intelligence’ towards much task-specific and real-world applicable measures of progress.” 

Measuring the “squishy” things

To assistance marque this shift, immoderate researchers are looking to the tools of societal science. A February presumption paper argued that “evaluating GenAI systems is simply a societal subject measurement challenge,” specifically unpacking however the validity systems utilized successful societal measurements tin beryllium applied to AI benchmarking. 

The authors, mostly employed by Microsoft’s probe subdivision but joined by academics from Stanford and the University of Michigan, constituent to the standards that societal scientists usage to measurement contested concepts similar ideology, democracy, and media bias. Applied to AI benchmarks, those aforesaid procedures could connection a mode to measurement concepts similar “reasoning” and “math proficiency” without slipping into hazy generalizations.

In the societal subject literature, it’s peculiarly important that metrics statesman with a rigorous explanation of the conception measured by the test. For instance, if the trial is to measurement however antiauthoritarian a nine is, it archetypal needs to found a explanation for a “democratic society” and past found questions that are applicable to that definition. 

To use this to a benchmark similar SWE-Bench, designers would request to acceptable speech the classical instrumentality learning approach, which is to cod programming problems from GitHub and make a strategy to validate answers arsenic existent oregon false. Instead, they’d archetypal request to specify what the benchmark aims to measurement (“ability to resoluteness flagged issues successful software,” for instance), interruption that into subskills (different types of problems oregon types of programme that the AI exemplary tin successfully process), and past yet assemble questions that accurately screen the antithetic subskills.

It’s a profound alteration from however AI researchers typically attack benchmarking—but for researchers similar Jacobs, a coauthor connected the February paper, that’s the full point. “There’s a mismatch betwixt what’s happening successful the tech manufacture and these tools from societal science,” she says. “We person decades and decades of reasoning astir however we privation to measurement these squishy things astir humans.”

Even though the thought has made a existent interaction successful the probe world, it’s been dilatory to power the mode AI companies are really utilizing benchmarks. 

The past 2 months person seen caller exemplary releases from OpenAI, Anthropic, Google, and Meta, and each of them thin heavy connected multiple-choice cognition benchmarks similar MMLU—the nonstop attack that validity researchers are trying to determination past. After all, exemplary releases are, for the astir part, inactive astir showing increases successful wide intelligence, and wide benchmarks proceed to beryllium utilized to backmost up those claims. 

For immoderate observers, that’s bully enough. Benchmarks, Wharton prof Ethan Mollick says, are “bad measures of things, but besides they’re what we’ve got.” He adds: “At the aforesaid time, the models are getting better. A batch of sins are forgiven by accelerated progress.”

For now, the industry’s long-standing absorption connected artificial wide quality seems to beryllium crowding retired a much focused validity-based approach. As agelong arsenic AI models tin support increasing successful wide intelligence, past circumstantial applications don’t look arsenic compelling—even if that leaves practitioners relying connected tools they nary longer afloat trust. 

“This is the tightrope we’re walking,” says Hugging Face’s Solaiman. “It’s excessively casual to propulsion the strategy out, but evaluations are truly adjuvant successful knowing our models, adjacent with these limitations.”

Russell Brandom is simply a freelance writer covering artificial intelligence. He lives successful Brooklyn with his woman and 2 cats.

This communicative was supported by a assistance from the Tarbell Center for AI Journalism.