Your AI problem is probably a documentation problem

Only 24% of respondents to our State of Docs survey track how fresh their documentation is.

That concerns me as a leader. We expect AI to explain our products and help customers use them, often drawing on documentation that changes whenever the product does. I’d want to know how we check whether that information is still current.

Freshness doesn’t guarantee accuracy, of course, but it’s one way to notice when content needs attention. Without that visibility, a team can miss problems in the context an AI tool relies on.

And with more than 60% of docs traffic now coming from agents, the quality bar is higher still. If an agent hits a missing step or has to choose between conflicting instructions, it’s the customer who suffers.

When scattered knowledge becomes a bad answer

Our own documentation had grown around years of product releases. Content lived across different spaces and workflows, and someone trying to publish their first site had to gather the instructions from multiple pages.

We knew how the pieces fit together. But we were asking a new user to work that out for themselves.

An AI assistant reading those pages faces a similar problem: it has to assemble an answer from scattered information. And its authoritative response will often make the source look more coherent than it really is. Red flags that a human might spot (a qualification elsewhere in a guide, or a clue that an instruction is old) can drop out when an assistant condenses several pages into one answer, and never reach the person asking the question.

Instead, that person gets what looks like a complete answer, and they may only discover what’s missing when they try to follow it.

The good news is that the opposite is also true: a clear explanation can help readers directly and support answers across many conversations. An AI assistant summarizing docs amplifies their quality, making good documentation great, and bad documentation worse.

Your documentation is part of your product experience

A customer who follows an assistant’s instructions and gets stuck experiences it as a failure of your product. Telling them the model accurately reproduced an outdated paragraph won’t help them finish what they were doing.

And that makes the outdated paragraph a concern for product leadership.

If I approve investment in an AI experience, I should also be asking who maintains the information behind it. Who owns the context? We can put considerable effort into model selection and retrieval, but then leave models to work from an unreliable account of our own product.

Those choices matter. So does the less glamorous work of checking and correcting the documentation. And I don’t think we can expect to deliver a good customer experience while leaving that work to chance.

Give that responsibility an owner

That’s why, in March, we hired Sarah Dugan as GitBook’s founding Docs Lead. One of her primary focus areas is improving our documentation and the process for maintaining it.

I’m conscious that people expect GitBook’s docs to be great. We expect it of ourselves. But over time, they had accumulated around releases until incremental fixes weren’t enough. Our docs weren’t wrong or out of date, but they no longer reflected how people were using GitBook after we’d added new features and AI workflows. Just because we’re a documentation company doesn’t mean we don’t struggle with the same problems our customers face.

The rebuild Sarah led gave us a more deliberate structure focused on user journeys, and a consistent review process. Before shipping the updates, the team asked our AI Assistant real user questions to check whether it could answer from the docs. The results showed a clear improvement in retrieval, and sharper answers.

To me, that's really encouraging, because the improvement came from work that also made the docs easier for people to follow.

But all of this work takes attention. Someone needs the time to examine the whole experience, the authority to question its structure, and the support to make changes. Hiring Sarah was an investment in giving our documentation that attention.

It also made the work after the rebuild clearer. The next product update can make a previously accurate page incomplete. I want us to be able to rely on our docs six months after publication, which means maintaining them with the same care we put into writing them.

Make problems visible, then help teams fix them

These challenges aren’t unique to us, and I want GitBook to help customers with that ongoing work, too. Once people and AI systems start using a published page, the docs team needs to understand where it’s falling short.

That's why AI agents reading docs on GitBook can now leave feedback when they hit a missing step or incorrect information. Someone can get an answer based on our documentation without ever visiting our docs site, so we need ways to hear when it fails them.

We’re starting with content gaps, link audit and style guide. In our own docs, these have identified knowledge gaps and shown us how often they recur, alongside broken links that need fixing. They give Sarah and the team a specific problem to investigate and fix.

GitBook Agent, or any agent connected through MCP, can help us prepare those fixes for review.

But I still want people reviewing those changes. A convincing AI rewrite of a docs page can still be wrong about the product, and someone needs to check it before publication.

These checks only cover part of documentation quality. Fixing every broken link wouldn’t tell us whether every instruction on a page was correct. But we’re building towards a broader system for keeping product knowledge reliable; content gaps, link audit and style guide are just the first steps.

Who hears when the knowledge fails?

I expect teams that treat their documentation as core infrastructure, and maintain it that way, will see their customers get more accurate answers from AI tools, and ultimately a better overall product experience.

This is what we’re constantly working towards at GitBook: documentation tools and practices that help our customers deliver high-quality, accurate answers to their users when they really need them.

But the question I’d take back to a leadership team is this: who owns the knowledge our customers’ AI agents rely on, and how do we hear when it lets someone down?

→ Read more about our new quality monitoring features

→ Research: AI agents are now the majority reader of your docs

→ Guide: How to audit documentation quality: a practical framework and checklist

→ How (and why) we rebuilt GitBook’s docs