A strict output format can block the reasoning a model needs to be right

Ask for clean JSON, get a worse answer

You can watch this happen with any capable chatbot. Give it an arithmetic word problem in plain language and it works through the steps and lands on the right number. Ask the same model for the answer as strict JSON, an object with an answer field and nothing else, wire that into a program, and the answers start coming back wrong more often. Anyone who has connected a model to code and told it to return only JSON has felt the quality slip.

In a 2024 study, researchers put a number on it: holding several models to a strict output format lowered their accuracy on grade-school math problems.¹ How large that cost is turns out to be the disputed part, and its size only comes into focus once you can see what the format does to the model as it writes.

The format is enforced one token at a time

Start with how a program pins a model's output to a shape at all. Left alone, the model produces its reply the way prefill-and-decode described: at each step it holds a probability spread over its whole vocabulary, every token it could write next with a weight on each, and it draws one. To force valid JSON, the program slips a step in between the spread and the draw. It builds a mask, a list marking every token that would keep the output valid and every token that would break it, and it drives the weight of the invalid ones to zero before the draw happens.² The model can only pick from what survives.

Watch a single step. You have asked for an object that starts with an answer field. At the very first token, the model's own spread might most favor Sure or Here: it was trained on prose, and prose is what it reaches for. But Sure is not the start of valid JSON, so the mask zeroes it. The only tokens left alive are the ones that can legally open the required object, an opening brace or a space. Whatever the model was leaning toward, it opens with a brace, because that is all the mask left standing.

This is settled engineering. Two independent implementations describe the same step. Willard and Louf compute the mask as a list of ones and zeros multiplied into the distribution before a token is sampled.² XGrammar builds a grammar for the required format and, at each step, sets every grammar-breaking token's weight to negative infinity, so it has no chance of being drawn.³

Providers ship this. OpenAI calls its schema-enforcing version Structured Outputs, and it works by exactly this masking: the company turns your schema into a grammar and holds the model to tokens that match.⁵ It saves the name JSON mode for a weaker, older setting that only promises the output is valid JSON, with no guarantee the fields are the ones you asked for.⁶ The two are not the same feature, and the names matter here, because the study we started with uses JSON mode to mean the hard-constrained kind. To keep them apart, the hard kind is called constrained decoding from here on, and JSON mode is kept for the weak one.

The blocked tokens were where the thinking went

Now the cost. Thinking-out-loud showed that a transformer does a fixed amount of computation per token: the only way it can spend more effort on a hard problem is to write intermediate steps and read them back. The scratch work is where the computation happens. Take away the model's room to write those steps and you take away the computation itself.

A strict schema can do exactly that. Picture a schema shaped {"answer": ..., "reason": ...}, the answer first and the reason second, which is a natural way to lay out a form. Once the model has written {"answer":, the mask will only let it write a valid answer value next. Every token that would begin a sentence of reasoning is blocked, because a sentence of reasoning is not a valid answer value. So the model commits to the answer before the reason field exists, before it has written a single step of the work that would have found the answer. The reason field, when it comes, is filled in after the fact.

This is not hypothetical. In the 2024 study, one model under the hard constraint put the answer key ahead of the reason key in 100% of its responses. Every time, the format marched the answer out first and collapsed the step-by-step working into a blind guess.¹ The mask did not make the model worse at math. It took away the space where the model did the math.

Two teams measured the cost and disagree

The mechanism is clear and the direction is clear: block the reasoning and accuracy falls. What is genuinely unsettled is the size, and the two teams that have measured it disagree sharply.

Tam and colleagues report large drops. On grade-school math, GPT-3.5-Turbo fell from about 76% correct in free text to 49% under the constraint, and Claude-3-Haiku fell from about 87% to 23%.¹ Taken at face value, those numbers say the format is enormously expensive.

The team behind Outlines, a constrained-decoding library, re-ran the same tasks and found the opposite: with the format enforced, their model matched or slightly beat its free-text score, 78% against 77% on the same math benchmark.⁷ They argue the earlier drops came from the setup, not the constraint. The prompts given to the constrained runs were not the same as the free-text prompts, and the schema forced the answer ahead of the reasoning, the exact foreclosure described above.⁷ This is a party with a stake, since they sell structured generation, and worth reading with that in mind. Their objection is still concrete and checkable.

And the case does not rest on dottxt alone. Constrained decoding is not always a tax. Earlier work found it can raise accuracy on some tasks by keeping a model from wandering off the required format.⁸ Tam's own results show format restriction helping on some classification tasks even as it hurt the reasoning ones.¹ The honest reading is that the mask itself is cheap. What is expensive is a schema that leaves the model no room to reason before it answers.

Let the model reason before it fills the form

The fix follows straight from the cause, and both camps agree on it: give the model room to reason before you force the format. Put a reasoning field first and the answer field last, so the schema asks for the work before it asks for the conclusion. Or let the model reason in plain prose and convert that to JSON in a second step. OpenAI's own documentation lays out a schema exactly this way, a steps array of reasoning followed by a final_answer field, as the recommended shape for a problem that needs thinking.⁶ The thing that matters is the order the schema asks for things in. A schema that reasons first costs little. A schema that answers first can cost a great deal.