Can an AI agent build an integration with your docs?

A developer asks an AI coding agent to build an integration. The agent connects to your documentation through MCP, finds the relevant instructions, and starts working.

But a few steps in, it needs an API key. Your docs explain where to enter it, but getting one requires access to a portal they never mention.

A developer reading the guide might recognize the missing step and contact support. An agent may ask its user for help, stop, or keep trying over and over, using credits with each attempt. You can’t count on a documentation gap producing a question your team will see.

And that means support conversations alone won’t tell you whether the docs are complete. You need to test what someone can build using the information you’ve published, then check whether it works.

For FreedomPay, that journey now largely happens through AI agents reading the docs via MCP. Its documentation rebuild began with a familiar problem: integration guides lived in PDFs across multiple spaces, and partners often needed one-on-one support to navigate them. John Lueders, VP of Developer Experience, wanted partners to find the information they needed and get to work independently. Making that information available to AI tools helps partners use it where they’re building.

To assess your own docs, pick one integration journey and follow it through these five stages. If partners use agents, test with those agents too.

Set up your test

Connect your agent to your published documentation through MCP. Choose a specific integration task and a test environment, then give it this instruction:

I’m testing whether our documentation supports this task: [describe the integration and intended result].

Use the connected documentation as your source for product-specific instructions. Cite the pages supporting your decisions. If information is missing or contradictory, identify it and explain what you need from me. Don’t invent product details or treat an assumption as a documented fact.

Work in [sandbox or test environment]

I’m testing whether our documentation supports this task: [describe the integration and intended result].

Use the connected documentation as your source for product-specific instructions. Cite the pages supporting your decisions. If information is missing or contradictory, identify it and explain what you need from me. Don’t invent product details or treat an assumption as a documented fact.

Work in [sandbox or test environment]

I’m testing whether our documentation supports this task: [describe the integration and intended result].

Use the connected documentation as your source for product-specific instructions. Cite the pages supporting your decisions. If information is missing or contradictory, identify it and explain what you need from me. Don’t invent product details or treat an assumption as a documented fact.

Work in [sandbox or test environment]

I’m testing whether our documentation supports this task: [describe the integration and intended result].

Use the connected documentation as your source for product-specific instructions. Cite the pages supporting your decisions. If information is missing or contradictory, identify it and explain what you need from me. Don’t invent product details or treat an assumption as a documented fact.

Work in [sandbox or test environment]

Once your experiment is set up, use the prompts at each stage below in sequence. Keep a record of any extra information you supply along the way: it probably belongs in your docs.

1. Find the right integration path

Make the use case clear

Imagine asking an agent to add a payments flow to an existing application. It needs to find the relevant guide and establish whether that approach supports your use case.

The guide should explain who it’s for, which environments it supports, and when to choose another approach. Similar guides need enough context to distinguish them, especially if they cover different product versions or integration methods.

A clear sidebar helps people browse. An agent retrieving documentation through MCP needs those distinctions explained in the content it receives.

Check what the agent retrieves

Give the agent a realistic task without supplying the page you already know contains the answer. Look at which documentation it retrieves and whether it chooses the right integration path.

If it selects an outdated guide, find out why. The current guide might be difficult to retrieve, or the older page might lack a clear explanation of its status. Each problem needs a different fix.

Keep checking routes into the published site as well. GitBook’s Broken links feature identifies incoming visits that return a 404 and suggests redirects where it finds a suitable destination. Reviewing those findings helps you repair failed routes into your docs alongside your agent retrieval tests.

Try this prompt

Find the documentation for [integration task]. Explain which approach fits [use case and environment]

Find the documentation for [integration task]. Explain which approach fits [use case and environment]

Find the documentation for [integration task]. Explain which approach fits [use case and environment]

Find the documentation for [integration task]. Explain which approach fits [use case and environment]

2. Understand what’s needed before starting

State prerequisites upfront

A guide should explain its prerequisites before anyone reaches the step that depends on them.

That includes accounts and credentials, along with permissions, supported versions, and work another team needs to complete. If sandbox access requires approval, explain how to request it and what happens next.

The agent needs enough context to work out which steps it can complete and which require the partner to act.

Test with a new partner’s access

Use the access and credentials a new partner would have. Your existing administrator account can hide problems: an endpoint might accept your credentials because you completed the setup years ago.

Watch what happens when the agent reaches a prerequisite it can’t satisfy. Does it explain what’s missing and ask the user to complete the required step? Or does it proceed as though access is already available?

Some integrations will involve a conversation or approval. Document the handoff so the partner knows when it’s required, who to contact, and what to provide. That also gives the agent the information it needs to explain the next step accurately.

Try this prompt

3. Complete a first working interaction

Define a result you can verify

Choose a small, meaningful milestone, such as making an authenticated request or completing a sandbox transaction.

The guide needs enough detail to run the example and recognize the result. It should explain which values to replace, where those values come from, and which environment to use. Include the expected response and explain what it confirms.

An API reference answers questions about individual endpoints. The integration guide should explain how those endpoints fit into the task, with links to the relevant reference entries.

Inspect the implementation

Ask the agent to complete the task from a clean setup, then check its work. It’s important to remember that a message saying the integration is complete doesn’t mean that it’s been completed correctly.

Review the generated code and run the relevant checks. Look for values or steps the agent introduced that your documentation doesn’t support. If the task failed, establish whether the necessary information was missing, wasn’t retrieved, or was retrieved but used incorrectly.

We used a related approach when rebuilding GitBook’s own docs, testing whether AI tools could answer real user questions from the documentation.

If you supply a missing detail during your test, be sure to record it and update the docs. The next partner’s agent who tries to complete the task won’t have your conversation as context.

Try this prompt

Use the documentation to implement [small, verifiable milestone]

Use the documentation to implement [small, verifiable milestone]

Use the documentation to implement [small, verifiable milestone]

Use the documentation to implement [small, verifiable milestone]

4. Recover when something goes wrong

Give errors an actionable explanation

Once the first interaction works, introduce a common failure. You could use an expired credential or leave out a required value, then try to recover using the documentation.

“Check your credentials” leaves a lot of work to the reader or agent. A useful troubleshooting entry explains where to find the relevant credential, how to check its environment and permissions, and when replacing it might help.

Does the agent connect the error to the right explanation? Its proposed fix should address the cause and preserve the integration’s requirements.

Look beyond support questions

Support conversations can reveal missing explanations, but they won’t capture every problem an agent encounters.

GitBook’s Content gaps compares questions from readers and information from other connected sources against your published docs, surfacing findings with evidence and recommended fixes. Your team can inspect that evidence and ask GitBook Agent to draft a change request.

Use those findings to choose failure cases to test, alongside problems you observe in agent-assisted builds. If an agent works around an error incorrectly and the developer never contacts support, your test may be where you first discover the gap.

When escalation is necessary, explain which error details or request identifiers to include and which credentials or sensitive data to exclude.

Try this prompt

In our test environment, [introduce a specific failure, such as using an expired test credential]

In our test environment, [introduce a specific failure, such as using an expired test credential]

In our test environment, [introduce a specific failure, such as using an expired test credential]

In our test environment, [introduce a specific failure, such as using an expired test credential]

5. Know whether the integration is ready to proceed

Separate a first result from production readiness

After getting a successful response, a partner needs to understand what remains before the integration is ready for production.

Depending on your product, that might include handling asynchronous events, testing failure cases, or completing a certification process. Spell out the relevant checks and explain what successful behavior looks like.

Keep sandbox and production differences easy to find. If partners need different credentials or configuration, document the transition where they’ll encounter it.

Check what remains unfinished

Ask the agent to use the docs to identify what still needs testing or approval. Have someone who knows the product review that answer against the implementation.

Look for requirements it omitted or claimed to have completed without evidence. The partner needs to know what works, what remains unverified, and which steps require their involvement.

Try this prompt

Record where the journey needs help

As you work through the five stages, keep a record of where the agent needed help or made an unsupported assumption. You can then use those observations to decide what to fix and how to test it again.

Review the evidence yourself

Treat the agent’s explanation as part of the test record. Inspect the cited pages and check the implementation against the product’s behavior. You should pay particular attention to claims of success without a recorded result, and missing information the agent didn’t flag.

Keep the retrieved documentation, any extra information you supplied, and the resulting implementation together. A human-led walkthrough is still useful for checking the instructions directly.

Here’s a simple worksheet to help you capture what happened and highlight issues:

*Record “verified,” “needed help,” “blocked,” or “incorrect result.”

Diagnose the gap before fixing it

Be specific about the problem. Writing “authentication is confusing” makes it difficult to assign, while writing “the guide asks for a sandbox key without explaining how to request one” gives someone a clear fix.

You also need to separate documentation problems from retrieval or implementation errors. Adding another explanation won’t necessarily help if the agent never retrieves the page that already contains it.

After publishing an update, repeat the affected task in a fresh session so the agent has to find the answer in your updated docs. Verify the result before marking the issue resolved.

Keep improving between reviews

A journey review captures one attempt with a particular set of tools and documentation. As your product changes and more partners build with it, a few regular habits can help you catch problems between tests:

• Review new content gaps and failed visits. GitBook’s Content gaps and Broken links give you evidence to prioritize issues that block progress. Assign each accepted fix to someone on your team. Both features are currently in beta on Ultimate sites; you can use the same process wherever you collect support questions and site errors.

• Check that guides use the same terms. If one says “client key” and another says “API key,” a reader or agent may treat them as different credentials. GitBook’s Style guide and Checks let you define conventions and check sections of your docs, with proposed fixes arriving in change requests for review.

• Ask someone who knows the product to verify technical changes before publishing. Review AI-generated documentation and integration code against the product’s actual behavior.

• Choose your next test based on product changes and observed failures. Work through the setup step that changed in the latest release, the error that keeps appearing in support, or an assumption you found in an agent’s implementation. Keep the results alongside your fixes so you can see whether the update resolved the problem.

On October 27, I’ll be talking with FreedomPay’s John Lueders about their documentation rebuild and how partners use it through MCP. Reserve your spot, and bring a stage from your own integration journey that you’d like to improve — we’ll save time for questions.

→ Join our webinar with FreedomPay on 27 October

→ How FreedomPay is rebuilding its integration experience with GitBook

→ Introducing GitBook’s quality monitoring layer for reliable docs