Improving your docs and advocating for them are two different jobs

There are two main reasons to measure documentation, and I think a lot of the frustration around docs metrics comes from not separating them.

The first is figuring out how to improve your docs: determining which pages are failing people and where the gaps are. The second is making the case for the work at all, to the people who decide where budget and headcount go.

I spent a couple of years conflating the two, and being quietly annoyed that my numbers weren’t very helpful for either improving the docs or getting buy-in. The same metric can be worthless for one and perfectly good for the other, and it’s important to recognize the difference.

Figuring out what to fix

At a company I worked at before GitBook, we looked at a couple of docs metrics and not much else. We would periodically review the count of page views, and time spent on page. But mostly we wrote guides when someone asked for one, or updated a page when we heard that something seemed confusing.

That’s not a bad approach, honestly, and it produced some pages I’m still proud of. But we were fishing in the dark. I had no real way of telling which parts were weak, or whether the things we shipped had helped.

That product was a data infrastructure tool which was complicated to use. Some of that complication was a product problem we were solving in the documentation, which is quite common. It had a lot of concepts that a user wasn’t going to pick up anywhere except the docs.

A line, not a point

Page views were never going to tell us whether the docs were working. A page with heavy traffic might be the thing everyone needs, or just the page that an error message links to. Time on page can be even harder to understand, because a long visit can mean either careful reading or being stuck. And there’s no way to tell from the outside.

What would have helped is looking at movement rather than position. A single month’s numbers tell you very little on their own. But if a section’s traffic slid steadily across a year, or a page changed shape after you reorganized the navigation, that at least tells you that something happened, and gives you somewhere to start asking why. Almost every quantitative signal is more useful as a line than as a point in time.

What people were already telling us

The other part of the picture was qualitative, and that took me longer to take seriously than it should have. We had a community with thousands of conversations across Slack and GitHub. After spending any serious time in those spaces, it was easy to see the issues which came up again and again: confusing concepts, and places people continued to get stuck.

There’s more of this type of feedback available now than there used to be. Search queries have always been worth reading, particularly the ones that return nothing. And if your docs have an AI assistant, the questions people type into it are a direct account of what they came for. Most of this is reachable with whatever analytics you already have, plus a regular conversation with your support team.

Making the case

Most documentation people I talk to have thought hard about whether their docs are good. But fewer have thought about how to make someone else care. I think that’s the more neglected skill by some distance.

The encouraging thing is that the bar is lower than you’d expect, and a lot of it comes down to framing rather than finding a better metric.

The same number in two rooms

Take page views. I’ve just said that they can’t tell you whether a page is working. But they’re a perfectly reasonable estimate of audience size. Saying that a few hundred thousand people used the documentation last quarter tells you nothing about whether any of it was good, but it does establish that the docs are load-bearing and widely used. For a budget conversation, that’s most of what you need.

It’s worth starting with the numbers you already have and asking what case they could support, rather than waiting until you have the perfect metric that proves everything. Doing that also highlights gaps in the metrics, and can help you understand which metrics might not be so far out of reach, like number of support tickets, or sign-ups that originated in your docs.

What your company thinks docs are for

The harder part is knowing what your company thinks documentation is for, because that determines which case will land. If docs are understood as a support function, the interesting number is the reduction of support tickets. If they’re a front door, it’s who arrives from search and what they do next.

In that job I mentioned earlier, the docs were how most people got started, so a docs failure showed up as a customer who never activated. That’s a much easier thing to walk into a budget meeting with than anything I could have measured on the docs site itself.

You won’t find the answer of what your docs are for in any tool you own. You might know it already, and if you don’t know it, there’s someone you can ask. Support knows which articles stop tickets, because they link to them several times a day. Sales knows which pages get forwarded. Whoever runs onboarding can list from memory where new customers get stuck. The information exists, but it’s distributed across other people’s brains and inboxes, and getting a hold of it means going and asking.

It’s worth doing that. Not only do you find out what evidence would actually persuade the people you need to persuade, you also end up with colleagues outside the documentation team who understand why the work matters. And when the case does get made, it’s much stronger coming from a few places at once than from you alone.

Somewhere to start

The thing I’d do first is take stock of what you already track, even if that’s minimal, and think about what each number could be for. Is this something you could use directly to improve the docs, or something that could justify their importance? There might not be a clear answer yet, and that’s fine — an empty pile is worth knowing about too.

Then pick a page you know well and look at a year of data, rather than a month. That might be page views, the search terms that brought people to it, or whatever support has said about that topic lately. None of those is convincing on its own, but reading them together might point at something fairly specific, such as a concept that gets introduced later than people need it. Knowing the page well is the part that makes this work — you need enough context to tell the difference between a real pattern and a coincidence.

Finally, go have that support or sales conversation and ask which page they send people to most often, and what people ask them that they could have looked up. I’ve gotten more out of fifteen minutes like that than out of another afternoon in analytics.

None of that would have needed better metrics than I already had back in that old job. I just needed a clearer sense of which question each number was answering, and a few more people who understood why the docs mattered.

I’m giving a talk that goes further into all of this at Write the Docs Berlin in September, including more detail on metrics you can track. If you have any thoughts on this piece, please feel free to drop me a line on the WTD Slack or on LinkedIn.