The function-calling score labs cite grades one call, and usually never runs it

A single percentage stands in for using tools well

When a lab says its model is good at using tools, the number behind the claim is usually a function-calling score, and the most-cited one comes from the Berkeley Function-Calling Leaderboard, built by the Gorilla team at UC Berkeley. On the board that ships with the team's 2025 paper, the top model, a version of GPT-4o, scores 66.4% overall.¹ The live board has since moved to a fourth version, and its own note defines that headline figure as an unweighted average across every sub-category it tests.²

What a reader wants that number to mean is that the model can be handed a set of tools and trusted to use them. It looks up an order, cancels it, issues the refund, and stops when the job is finished. That is a fair thing to want to know. It is not what the score measures. A BFCL number grades one thing at a time, a single proposed call, checked against a known-correct answer.

A model does not run the tools it is given. It writes a request, in structured text, that some other program runs on its behalf, a mechanic that is its own lesson. BFCL grades that written request. The question it answers is whether the model wrote the right call, not whether calling it accomplished anything.

How one call gets graded: parse, run, or decline

BFCL grades a proposed call in one of three ways, and which one applies depends on the category of the test case.³ The single-turn cases come in five shapes: one call, a choice among several offered tools, several calls in parallel, both at once, and a test of whether the model declines when nothing fits.³⁴

The most common grading method is Abstract Syntax Tree matching, or AST matching. An abstract syntax tree is just the parsed structure of a call, its function name and its arguments pulled apart into labeled pieces. The grader reads the model's proposed call into that structure and checks the pieces against the reference without running anything. The rule the paper states is exact about one point that is easy to miss: a call is correct "if the function name matches exactly and if all parameter values fall within their respective possible answers."¹ The name has to be right to the character. Each argument, though, is checked against a set of acceptable values rather than a single one.

Say the reference answer for "what is the weather in Paris, in Celsius" is get_weather(city="Paris", units="metric"). A model that writes units="celsius" can still be graded correct, as long as the test lists both "metric" and "celsius" among the acceptable values for that argument. Whether the call passes turns on whether each value lands inside the set the test authors wrote, so that set is doing as much of the work as the model is.

A smaller set of categories is graded by executable evaluation: the grader actually runs the proposed call and inspects what comes back. "Execution involves running the specified function and examining its output."³ The output is compared exactly for most results, within a 20% band for numbers that change in real time, and by shape and keys alone for nested results.³ A third category tests the opposite skill. The model is handed functions that do not fit the question and is scored on whether it correctly declines to call any of them.³ A model that always reaches for a tool scores zero here, however good its calls are everywhere else.

The split between the first row and the second matters more than it looks. In the first version of the board, of roughly 2,000 test cases, only about 300 were run, and the rest were parsed.³ The live set the team added next, 2,251 real user queries, is graded by AST matching alone.⁵ So the largest slice of a function-calling score is a grader reading a proposed call, not watching it work.

The grade is faithful to the call it checks

None of this makes AST matching loose. The team checked it against the harder method: on the single-turn set, the scores from parsing calls track the scores from actually running them closely enough that the paper reports the two as strongly correlated.¹ Reading a call and running it reach nearly the same verdict on whether that one call was right. The limit of a function-calling score is not accuracy. It is reach. A faithful grade of one call is still a grade of one call, and an agent is never asked to make exactly one.

From one graded call to a whole task

An agent chains calls. It reads what each one returns, adjusts, and decides when the job is done. The paper's own top score, 66.4%, is already dragged down from the near-perfect single-call numbers. The categories that ask for exactly this behavior are where the best models collapse toward zero on holding facts in memory across a task.¹ The board's results show the same shape: the top models ace the one-shot questions and stumble once they must remember context, manage a long conversation, or decide not to act.¹ The team itself is candid about the split.

The sister lesson on τ-bench put a number on that gap using the same native function calling BFCL grades. The same GPT-4o agent solves about 61% of τ-bench's retail tasks on a single try. Asked to complete the same task eight times running, it succeeds under a quarter of the time.⁶ The single-call skill is genuine. It does not add up to a reliable agent on its own.

BFCL is not ignoring this. Its later versions exist to reach past the single call. Version three, released in September 2024, scores a whole multi-turn conversation, and the fourth adds agentic tasks such as memory and search.⁷ So far those categories expose the gap rather than close it. The multi-turn set is young and small, a thousand entries against the four thousand-plus single-turn and live cases,⁷ and its scoring metric was overhauled six weeks after it launched.⁸ The score that reads as agent reliability is still mostly the score of one call.

When the leaderboard rewrote its own answers

There is one more reason to hold the narrow number loosely: the board has repeatedly rewritten the answers it grades against.⁸ The team's own changelog records fix after fix to mislabeled and buggy test cases. The largest single entry, dated 16 October 2024, corrected 547 of the roughly 1,053 cases in the live "multiple" category, along with a hundred-odd others in the same fix. That is more than half of one of the board's largest categories.⁸

Every score published against that category before the fix was graded against answers the team later called bugs. And the changelog records what was corrected, not how the corrections moved the board. A lab that chose one model over another on the strength of a live "multiple" score in early 2024 has no published way to check that call. It cannot know whether the fixed data would have ranked the two the same. The number was cited with confidence while its answer key was still being rewritten.