Summary
We've been thinking a lot about docs quality at GitBook, and I really wanted to know how to quantify the effects of different documentation defects. So I built a fictional API, wrote correct docs for it, and then broke the docs four different ways — a dead link, an inconsistent field name, a value documented backwards, and a required step left out entirely — and gave each version to two AI coding agents.
The missing step was the most expensive one: agents burned up to ten times the normal cost trying to get past it. The backwards value cost nothing extra, but every agent that hit it produced the wrong result and reported success.
This is the second project in our series on how AI coding agents use API documentation. The first, TomatoPy, asked whether agents read the docs at all. This one asks what happens when the docs are wrong.
What we found
Drift is the dangerous docs defect. A value documented inconsistently from the API cost nothing extra, but every run — 12 of 12 — cooked the wrong risotto and reported success.
Content gaps are the expensive docs defect. Both agents failed and were aware of it, but Claude Code brute-forced the API first, at 2x to nearly 10x the normal cost and up to 1,356 API calls.
The gap reached the docs assistant. All 24 runs that hit the missing step asked the docs AI assistant, and it confirmed, from the same incomplete docs, that nothing was missing.
Broken links and inconsistent names cost almost nothing.
llms-full.txtabsorbed the dead link, and the API's own response fixed the field name.Agents trusted the docs over their own knowledge. Told to toast rice for 30 seconds, 12 of 12 runs did, against what every model already knows about risotto.
Agents can guess the conventions, but not your product. Without docs, Claude Code hit the 15-minute cap at 4/16, six times out of six. Codex reached 11/16 from REST conventions and cooking knowledge. No run got past the serenade step or the texture scale.
Background
Proving the value of docs came up in nearly every conversation I had for this year’s State of Docs report, and so did the other side of it: everyone knows bad docs are expensive, but nobody can say what they cost.
With human readers, that cost is harder to define. A developer may hit a broken link, mutter, and find the page another way, without anyone logging the minutes. A missing step may turn into a support ticket a week later, filed by someone else. But when the reader is an AI coding agent, the cost has an impact you can actually add up: tokens, wall-clock seconds, API calls, and a script that either works or doesn’t.
In my previous TomatoPy experiment, I gave three coding agents a fictional pizza-making API and correct documentation, and watched how they read it. One big finding from that experiment was that the API’s error messages did a lot of the docs’ job: when an agent skipped the reference page and sent a bad request, the 422 response named the missing field, and the agent fixed its code from the error. But real APIs are often less helpful - not every API has descriptive or useful error messages. And because this API was generated by an agent and was quite contained, it was complete in a way that some human-authored APIs might not be.
In this experiment, I wanted to ask a different question, which feels increasingly urgent: what does a defect in the docs actually do to an agent, when the API won’t bail it out?
So this time I built an API with less helpful error messages, made the docs the only place the rules live, and then broke the docs on purpose.
If you already know the setup and want the numbers, jump to the results →
The setup: a fictional risotto API, documented on GitBook
RESTotto is a fictional REST API for making risotto. You light a stove, create a pot, toast the rice, ladle in broth, stir, and plate. Food tastes better when it’s cooked with love, so RESTotto also requires you to serenade the pot before the first stir. That step has no real-world equivalent, so the only way an agent can know about it is from the docs. (If you’d like to make the analog version, this one is a good place to start. Singing to it is optional.)
I like using fixtures like this for experiments, because it’s realistic in construction, with API keys, session headers and typed JSON bodies, but absurd enough in subject that no model arrives with any idea how it works. I couldn’t reuse PizzaStack, the API I used for TomatoPy: it’s public now, and so could be in somebody’s training data by this point.
The RESTotto API runs on a plain hostname that gives nothing away. The task is identical in every run:
You are a developer using the RESTotto API for the first time. The documentation starts at [URL]. Write and run a Python script that uses the RESTotto API to make a risotto: light a stove on medium heat, create a pot with 320 g of carnaroli rice, toast the rice, cook it with vegetable broth, stir it until it is creamy and loose (all’onda), and plate it. Print the tasting card. [...] When you are finished, end your final message with exactly one of these lines: RESULT: SUCCESS / RESULT: FAILURE
The docs live on GitBook. The prompt names the target texture (all'onda) in words, but never as a number: that number only exists in the docs.
How the API was designed so only the docs had the answers
Taking what I learned from the TomatoPy experiment, I applied three principles when designing the RESTotto API.
Every error looks the same. Any invalid request — missing field, wrong field name, out-of-range value, steps in the wrong order — returns 400 {"error": "invalid request"}. There is no auto-generated schema page, no /openapi.json, and GET / returns only the API’s name. An agent can’t automatically fix its code by reading the error.
The tasting card is a plain echo. When you plate your programmatic risotto, the response lists the numbers you sent — toast seconds, ladle sizes, stir minutes, texture — and nothing else. No adjectives or description. The only way of determining whether those numbers were right is looking at the docs.
Scoring is server-side and silent. The API logs every request. After each run, a scorer grades the agent’s final pot on sixteen criteria: six for completing each step, two for order, three for following the prompt (medium heat, 320 g carnaroli, vegetable broth), and five for rules that exist only in the docs (toast time, ladle size, total broth, stir time, and the texture value). The agent doesn’t see the score — that’s calculated in the harness outside of the agent’s purview.
The seven docs versions: control, four defects, combined, and no docs
The crux of the experiment is comparing different versions of the docs to understand the cost of different docs issues. Every version is generated from the same correct source by a script that applies small, named edits, so a variant can’t drift from the control by accident. Each lives at its own GitBook site with an opaque URL so that the agent can’t search to find what it needs on a correct version of the docs site. You can see the docs for the control group (no defects) here.
Version | What’s wrong | Where it breaks |
|---|---|---|
Control | Nothing | — |
Broken inbound link | The run starts at an old quickstart URL that returns a 404. The page still exists at its new address, in the nav, the sitemap and | Before step 1 |
Terminology mismatch | The Stirring page and the reference call the pot’s ID | Step 5, stir |
Drift | The Stirring page documents the texture scale backwards: 1 = all’onda, 5 = stiff. The API uses the opposite, and accepts any value from 1 to 5 without complaint. | Step 5, stir |
Content gap | A required step is missing from the docs. Before the first stir, you must call | Step 5, stir |
Combined | One defect per step: the dead start URL, | Start, 2, 3, 5 |
No docs | No docs site. The prompt gives the API’s base URL instead. | Everywhere |
The three single defects that touch content (terminology mismatch, drift, and content gap) all sit at the same step, so their costs are comparable. The content gap is a true absence: serenade is not a word an agent can guess from cooking. The drift is purposely designed to be silent — the idea is that the API accepts the wrong number and doesn’t know of its mistake without the benefit of docs or human judgement.
Methodology: Two agent harnesses, 84 runs
I ran Claude Code on Claude Opus 5.5 and Codex CLI on GPT-6.1 Sol, both at medium reasoning effort. Each agent ran every docs version six times: 84 runs in total, on October 4, 6 and 8, 2026.
Every run was:
Headless and unattended
In a fresh workspace with a fresh agent config
Using a fresh Python with nothing installed, and a fresh
HOME.
No memory or history carried over. Runs were shuffled so no variant lined up with a time of day, and capped at 15 minutes.
How cost was calculated
Cost is at API list prices for both agents: Claude Code’s own cost report, and Codex’s token counts priced per the published rates. Claude Code started every run on Opus 5.5; in ten long runs its safeguards switched it to Opus 4.8 partway through, and those turns are priced at Opus 4.8’s rate (more on this in the caveats section).
Any run that reached another variant’s docs site or read the experiment’s own repo would have been excluded, but none did.
Results: cost and outcome for each docs version
With correct docs, both agents got a perfect 16/16 every time. Claude Code took a median of 38 seconds and $0.24 per run; Codex took 61 seconds and $0.09. Those are the baselines everything below is measured against. (Codex is cheaper because its model is priced lower per token; it actually made twice as many tool calls.)
Here are the results. The score measures the agent’s success in cooking perfect risotto. It is out of 16 and was identical across nearly every run of each version across both Codex and Claude Code; only no-docs splits by agent. Cost and time are medians relative to the same agent’s control, with the range across runs in parentheses.
Some notes:
Drift cost nothing extra in tokens, but was the only defect that produced a wrong result the agent didn’t notice. On every metric I tracked except the success of the dish itself, drift runs are indistinguishable from control.
The ranking by cost and the ranking by outcome don’t match. In the Drift case, the agent reports complete success, but didn’t realize that it is actually missing one criterion. In the Combined case with all four docs defects, the overall score was worse, but the time was better than that of the Content gap case.
The content gap was the defect that cost the most, and it never fooled the agent. All twelve runs reported failure, correctly. The bill came from how hard the agent tried before giving up, and the two agents tried very differently: Codex tried a handful of request variations and stopped at 1.8x control. Claude Code brute-forced the API — up to 1,356 calls and 40 test pots in one run — and its bill ranged from 2x to nearly 10x.
Something that surprised me: Two defects that docs teams spend a lot of time chasing — broken links and inconsistent names — cost agents almost nothing. Both were absorbed by something other than the docs page: the dead link by
llms-full.txt, which GitBook publishes for every site, and the wrong field name by the API itself, whose create-pot response already used the right one (more on both below).
Drift: an imperfect result reported as success
The drift defect models what happens when an API is updated but the docs stay the same. The docs in this case have one wrong page — the Stirring page describes the texture scale backwards: the docs say 1 is all’onda, while in the API all’onda is 5. The API accepts any value from 1 to 5 without complaint.
All twelve runs read the Stirring page, sent texture: 1, got a 200, plated the pot, and ended with RESULT: SUCCESS. And all twelve scored 15/16 — everything was perfect except the one thing the docs got wrong. The median cost was within $0.01 of control for both agents and the ranges overlap; drift runs took a few seconds longer for Claude Code and about 13 seconds longer for Codex. From the outside, a drift run looks just like a good one.
It’s interesting to read the agents’ final reports for what they treat as verification. Codex wrote, in several runs, that the tasting card’s texture: 1 “confirms all’onda”. Claude Code explained its reasoning: on the docs’ scale, 1 is “loose, flows like a wave”, which matches what was asked for, and the scale’s separate “creamy” value is 2, so it read “creamy and loose” as 1.
One detail worth noting: I ran a pilot before running the full experiment, and in that earlier version of the API, the tasting card described the finished texture in words rather than just echoing the number. With that card, both agents noticed the mismatch between what they’d asked for and what came back, cooked a second pot with the right value, and finished 16/16. This suggests where the danger is: drift is silent when the API gives the agent nothing to check against, but good API responses can mitigate this. This was a finding of the TomatoPy experiment too.
Content gap: every run failed, at up to nearly 10x the cost
Because this experiment used an API that was based off of a real cooking process, I included an extra step which didn’t quite mirror the real world steps and couldn’t be guessed. This was “serenade” in which you’d have to sing to the risotto (everyone knows that the best food is made with love).
The content gap case removes the serenade step entirely. It’s not on the Stirring page or reference, and therefore not in llms-full.txt. The first stir returns 400 {"error": "invalid request"}, and so does every stir after it.
No run managed to get past this — all twelve out of twelve ended with RESULT: FAILURE. In this case, the agents were aware of the failure — none plated an unstirred pot and called it done. In fact, several Claude Code runs explicitly declined to, because “that wouldn’t be the risotto you asked for”. Both agents were straight about this, but they did diverge here in cost.
Codex
Codex’s content gap runs cost 1.8x control (range 1.6–2.0x) and took about two and a half minutes. It re-read the Stirring page, tried a handful of variations on the request, confirmed plating also failed, and reported. Its runs made 15 to 35 API calls against 10 for control.
Claude Code
Claude Code’s content gap runs cost a median of 4.8x control, with a range from 2.0x to 9.7x. The most expensive run cost $2.29, made 1,356 API calls, and created 40 throwaway pots to test its theories against. Another made 1,036 calls on 34 pots. These runs tried every field name, type, encoding and HTTP method they could think of, fuzzed the request body, wrote probe scripts, and queried the pot’s state between attempts. Several reported afterwards, accurately, that they’d left “about 25 small unplated test pots” in the session. That’s a lot of unfinished programmatic risotto!
You may have noticed that the spread across runs here was quite wide. Two of the six runs gave up at 2.0x and 2.9x, close to Codex. The other four kept going, and in these runs Claude Code actually switched models mid-run (see the caveats). Whether long probing triggered the switch or the switch prolonged the probing, I can’t tell from the transcripts. Either way, a missing step turned a 38-second task into a brute-force search of the API, with up to 40 test pots and 1,356 calls.
The AI docs assistant confirmed nothing was missing
A missing step doesn’t stay on the page it’s missing from. It propagates into every surface that answers from the docs – an in-site assistant, an MCP server, or a chatbot trained on the content – and comes out the other side as a confident “no, nothing else is required”. In this experiment, that surface was GitBook’s AI docs assistant.
GitBook appends a short block of agent instructions to each page’s Markdown version, explaining that an agent can ask the documentation a question by adding ?ask= to any page URL. Every one of the 24 content gap and combined runs, across both agents, made use of this once stirring failed.
The questions were good ones. “Why does POST /pots/stir with texture 5 return 400? Are there extra requirements or fields?” “What are all the preconditions for POST /pots/stir to succeed? Must anything happen before stirring?” “Is there a changelog or known issue with the stir endpoint?”
The answers were confident, well-formatted, and wrong. The assistant listed the three documented fields, confirmed the values were valid, suggested checking the session ID and the JSON types, and in several cases stated plainly that no extra requirement for texture 5 is documented. If you’ve spent a lot of time with agents, you’ve probably experienced an agent’s absolute confidence in the face of being totally wrong. But the assistant answered from the docs, and the docs were missing the step — it couldn’t know what wasn’t there.
The agents treated the assistant’s answer as a second opinion that ruled the docs out. One Claude Code run put it this way: the assistant had confirmed that its request matched the spec, so the problem had to be a defect in the API’s stir endpoint, not the request. Then it told the user its script was correct and would finish the dish once the endpoint was fixed. Other runs reached the same conclusion with less certainty — “either the stir endpoint is broken, or it needs something the docs don’t mention” — but none of the 24 landed on the actual cause. They couldn’t have.
Combined: the content gap decided the outcome, and agents still followed the docs
The combined version has one defect per step: the dead start URL, pot_id in the quick-start, a toast time documented as 20–45 seconds (the API wants 60–180, and accepts anything), and no serenade. I wanted to know whether defects compound, and whether an agent that has already found two problems in the docs starts trusting them less.
On outcome, the combined version behaved like the content gap, because the content gap was in it. All twelve runs stopped at the stir step and reported failure. The broken link and the naming mismatch were absorbed on the way, as in the single-defect versions.
On cost, it was cheaper than the content gap alone for Claude Code, 2.1x against 4.8x, and about the same for Codex. I don’t think more defects made the agents more efficient. Claude Code’s combined runs made far fewer API calls than its content gap runs (a median of 105 against 374), four of six read the whole site through llms-full.txt after the dead link, and none switched models. My reading is that this is about how soon Claude Code gave up, not about efficiency: its six combined runs all cost between $0.42 and $0.67, about where its two cheapest content gap runs landed ($0.47 and $0.70), and none went on to the long probing that pulled the content gap median up.
The docs in this version also said to toast the rice for 20 to 45 seconds. Conventional wisdom when making real-life risotto says that you’re supposed to toast for a minute or two. Interestingly, in the no-docs runs, Codex got 90 seconds right from general knowledge from its training data alone (see below). But in all twelve combined runs, both agents toasted for 30 seconds, the documented value, and lost a point from their score. Having just worked around a broken link and a mislabeled field, neither agent second-guessed a number that contradicted its own knowledge.
Terminology and broken links: almost no extra cost
Terminology mismatch: agents took the right field name from the API’s response
The terminology defect is one field with two names. The Stirring page and the reference call the pot’s ID pot_id; the create-pot response, and the rest of the docs, call it vessel_id. Nothing in the docs connects the two, and the API only accepts vessel_id.
Every run finished 16/16. Eleven of the twelve hit at least one 400, almost always on the first stir, then looked at the create-pot response they already had, which said vessel_id, and switched. Three of the six Claude Code runs mentioned the discrepancy in their final report. Claude Code’s median cost was indistinguishable from control (0.9x); Codex spent about 30% more (1.3x), mostly on extra reads and a retry.
This defect was cheap because the right name was on the agent’s screen the whole time. A terminology mismatch where neither name appears in any response — two synonyms in prose, say — would be a different experiment.
Broken inbound link: agents read llms-full.txt instead
The broken-link runs start at an old quick-start URL that returns a 404. The page they wanted still existed at /quickstart, in the nav, in the sitemap, and in llms.txt.
In the experiment, this defect cost nothing. All twelve runs finished 16/16 right around the control cost. Both agents, in every run, went from the 404 directly to llms-full.txt — the file GitBook publishes with the whole site’s content in one document — and read the docs from there. Three of the twelve runs found it through llms.txt first; the rest requested it directly, which means they assumed that GitBook sites have one. For Claude Code, the broken-link runs were slightly cheaper than control (0.9x), because one large read replaced nine or ten page fetches.
Both agents told the user about the dead link. Codex said so mid-run in all six runs (“The supplied link returns 404, but I found the current quick start”). Claude Code mentioned it in its final report in four of six.
This is a finding about coding agents on a docs platform that publishes a full-site text file at a predictable address. It says nothing about what a dead link costs a human developer who lands on the same 404 from a blog post, or an agent landing on a docs site without an llms-full.txt, and it depends on the agent already knowing the convention.
No docs: agents guessed the conventional steps, but not the product-specific ones
I ran a baseline with no documentation at all, just the API’s base URL, to see how much of the task an agent can get right from conventions and general knowledge. And it did better than I expected, though still not adequate.
Codex scored 11/16 in all six runs. It guessed /stoves, /pots, /pots/toast and /pots/ladle from REST conventions and the task description, worked out the field names by probing, and shockingly, got three docs-only rules right based on cooking knowledge from their training data: a 90-second toast, 240 ml ladles, and 960 ml of broth for 320 g of rice. Then it hit the stir step, which needs the artificial “serenade” step and a texture scale it couldn’t guess, tried for a while, and reported failure at 3.9x control cost.
Claude Code scored 4/16 in all six runs, and all six hit the 15-minute cap at around 11x control cost; one run made 32,864 API calls. It found the stove and the pot, then spent the rest of its time fuzzing for the toast endpoint and never found it. Its last messages before the cap were continuing to propose new search strategies.
An agent can reconstruct the conventional parts of an API, and a fair amount of the domain knowledge, without being told. What it can’t reconstruct is what’s specific to your product: a step with an invented name, or a scale that runs in the direction you chose. Codex got further without docs than I expected, but it stopped at exactly the point where RESTotto stops behaving like other REST APIs. That’s the part where the docs were necessary, and it’s where a gap or a wrong value costs the most.
Those are also the parts I’d expect to go missing or wrong if you auto-generate your docs with AI. Promptless’s DoGBench benchmark, published at the end of September, points the same way: in an audit of 1,267 documentation patches written by coding agents, 45.5% had a gap that would stop a reader completing the task, including 21% that left out a prerequisite, and 36.6% misstated how something behaves. A third of the failures came from agents filling missing evidence with plausible assumptions — which is exactly what goes wrong when your product doesn’t perfectly follow conventions.
Agents trusted the docs over both the API and their own knowledge
If you put the findings side by side, a pattern comes out that I think matters more than any single cost multiple.
In 72 runs with docs, no agent ever doubted them. The docs said texture 1, so the agent sent 1 and checked the echo against the same page. The docs said toast for 30 seconds, so it toasted for 30 seconds, against everything it knows about rice. And when the docs omitted a step and the API refused the request, neither agent concluded that the docs were incomplete. They concluded that the API was broken, asked the docs assistant to confirm, got a confirmation drawn from the same incomplete source, and told the user the script was correct.
This is a testament to the importance of docs. It’s the behavior I’d expect from an agent reading good documentation, and it's exactly what makes bad documentation dangerous. A human developer who hits a 400 on a documented request will, at some point, wonder if the docs are missing something. Both agents, across 24 runs, tested the opposite hypothesis instead: that the docs were right and the API was wrong. In Claude Code’s case, this was tested very thoroughly at 1,356 API calls.
The two agents failed differently from each other. Codex/GPT-6.1 Sol gave up early and cheaply; its reports were short, factual and correct about what happened. Claude Code/Opus 5.5 kept going, spent more, and wrote a longer report that was also correct about what happened and more confident about why. Again, neither style caught the gap, because the information wasn’t there to catch, but the difference between them was a $0.16 report vs. a $2.29 one.
Drift is the one defect neither agent could detect, and it’s also the cheapest, financially, to hit. It scored 15/16 and reported success, twelve times out of twelve. If you’re a docs team deciding what to fix first, the cost table above says content gaps are expensive. But the outcome column says drift is the one you’ll never hear about until something fails further down the line.
Caveats and limitations
The multiples are rough. Six runs per cell, one API, two agents. The ranking of defects is stable across every run; the exact multiples are not, and the content gap range for Claude Code (2.0x to 9.7x) shows how much one agent’s behavior can vary on the same input. Treat the medians as order-of-magnitude, and the ranges as part of the result.
Claude Code did not stay on one model. Every run started on Opus 5.5. In ten of its 42 runs — four content gap runs and all six no-docs runs — Claude Code’s safeguards switched the model to Opus 4.8 partway through, after the agent’s API probing was flagged by an automatic check. The affected turns are priced at Opus 4.8’s rate, so the dollar figures are right, but those runs are not pure Opus 5.5 runs. The switch happened only in the longest, most probe-heavy runs, and I can’t say whether it changed their cost. The combined runs never triggered it, which is part of why combined came out cheaper than content gap.
The two agents aren’t a model comparison. Claude Code and Codex differ in harness, pricing, default tools and how they decide to stop. Codex’s lower cost reflects a cheaper model as much as fewer actions. This report is about how two agents behaved, and I wouldn’t read a ranking of the models into it.
This is silent drift. The drift defect was specifically designed so that the API would accept the documented value without the response contradicting it. Drift that produces an error may behave more like the terminology mismatch: cheap and visible, with the agents more easily recovering from it.
The content gap was unguessable by design. I invented the serenade step so that no amount of cooking knowledge could fill it. Real gaps are often more guessable than that, and the no-docs baseline shows that agents can guess a surprising amount. The cost of a gap in practice will depend on how far it is from what the agent already expects.
The broken-link result depends on GitBook conventions. Both agents already knew to look for llms-full.txt. On a docs platform that doesn’t publish one, or with an agent that doesn’t know the convention, a dead inbound link would cost more, and it costs humans something regardless.
No humans. This experiment specifically measured agents. I didn’t run the same defects past developers, so I can’t say how the costs compare — only that for humans, the broken link and the terminology mismatch would probably not have been free.
What this means for docs teams
Hunt for the defects that don’t produce errors. Broken links and inconsistent names announce themselves in obvious ways, but a value that’s wrong but valid doesn’t. If you have a parameter whose scale, direction or units changed at some point, that page deserves more scrutiny than any link checker will give it. Regular tests of the docs, either manual or automated, will help here.
A gap on one page is a gap everywhere. Missing or erroneous information in your docs will propagate. Any assistant, chatbot or MCP server that answers from your docs will answer with the gap in it, and will do so with the same confidence it brings to everything else. Agents treat that answer as verification. This is also why we built content gap detection into GitBook — a gap is cheaper to find before an assistant starts answering from it. When you add a step to a product, you’re not just adding it to your doc, but to everything downstream of the docs as well.
Help the API catch what the docs get wrong. This is the lesson from the pilot and the TomatoPy experiment: when the API includes helpful responses, an agent can see the mismatch and address it. A response that says only what you sent can’t contradict a wrong value. Where you can, make success responses mean something, not just echo.
Publish a full-site text file, and expect agents to use it. llms-full.txt turned a dead inbound link into a non-event for agents, and they went looking for it without being told. GitBook generates llms.txt and llms-full.txt for every published site automatically, which is why the agents found one without being told. This is cheap insurance against every navigation defect that you haven’t found yet. But it doesn’t help humans reading your docs, so remember to fix the link as well.
Budget for the brute force. When an agent can’t complete a task from your docs, it might not fail quickly. It will keep searching and trying until it figures out what it needs, and that can get expensive. For one of the two agents, a single missing step multiplied the bill by five, and in the worst run by nearly ten. If your users are running agents against your API at scale, your content gaps could be on their invoice.
The full benchmark — the API, the docs build script that generates every variant from one source, the harness, the scorer and the analysis script — is on GitHub at github.com/GitbookIO/RESTotto if you want to run it against your own agents. The docs sites stay up. The API requires a key to keep the environment stable.
