Paying Users to Test Your README: A Developer's UX Wake-Up Call

When "RTFM" Fails: The Hidden Cost of Poor Documentation

Developers have a long-standing reputation for neglecting documentation. The acronym "RTFM" (Read The F***ing Manual) is thrown around liberally, yet the manual itself is often little more than a collection of half-written notes, outdated tips, and links to abandoned forums. This is a disservice to users, and it's a problem that Terence Eden, a well-known developer and blogger, decided to tackle head-on.

In a recent blog post, Eden details a radical experiment: he paid volunteers to attempt to follow his project's README, then used their feedback to improve it. The results were both humbling and illuminating, offering a powerful lesson for the entire developer community.

The Experiment: €25 for an Hour of Brutal Honesty

Eden's project, ActivityBot, is an ActivityPub bot. As part of his NLnet grant application, he committed to testing the installation experience with real users, offering €25 for an hour of their time. He put out a call on Mastodon, gathered a handful of participants, and conducted one-on-one video calls.

The format was simple but effective. Eden asked each volunteer to share their screen and speak aloud their thought process as they followed the README. He took handwritten notes, deliberately avoiding the use of AI transcription tools, and listened as his carefully crafted instructions were deconstructed in real-time.

Devastating Findings: From Broken Links to Confusing Jokes

The feedback Eden received was a masterclass in humility. The list of issues discovered was extensive and, in many cases, painfully obvious in hindsight:

• Broken demo link: The link to the demo tool was incorrect, a fundamental error that would immediately derail any user.
• Terminal vs. GUI confusion: Some users read the README directly in the terminal, which affected how they interpreted formatting and commands.
• Ambiguous instructions: Instructions like "rename the file" were unclear—did that mean using the mv command, a GUI, or something else? Even renaming hidden files proved to be a point of confusion.
• Missing context: Eden hadn't actually explained what the software did or why a user would want to install it, leaving testers in the dark about the project's purpose.
• Poor section ordering: The flow of the document was confusing, with sections appearing in an illogical order.
• Inaccessible humor: His jokes, which Eden thought were lighthearted, were actively confusing to readers who took them literally.
• Unexplained jargon: Technical terminology was used without explanation, assuming a level of knowledge not all users possess.
• Overly technical sections: A whole section that was technically interesting to Eden was utterly confusing to everyone else, serving no practical purpose for the end-user.
• Platform-specific assumptions: Not all web servers handle permissions the same way, and the README didn't account for this variability.

Iterative Improvement: The Key to Effective Docs

Rather than being discouraged, Eden used this feedback to fuel an iterative improvement process. After each session, he updated the README based on the specific difficulties the tester encountered, then presented the revised version to the next participant. This cycle of test, refine, and re-test proved incredibly effective.

In total, Eden spent around €150, which he wryly notes is "cheaper than therapy." The investment yielded a demonstrably better README, one that is now easier for a new user to follow. While he acknowledges the document is far from perfect, the improvement is tangible.

Why Not Just Ask an LLM?

In an age where large language models are increasingly used to simulate user behavior, Eden's approach might seem antiquated. But he offers a compelling defense of the human touch. "People are brilliant!" he writes. "They can make you laugh, you can see their cat when it wanders on to the call, they bring a unique perspective to the problem, and they're really happy when you give them a €25 voucher."

More importantly, an LLM cannot replicate the visceral feedback of hearing a user's frustration in their voice. It cannot ask clarifying questions or react to unexpected discoveries. As Eden points out, drawing on his experience writing for GOV.UK, a second pair of human eyes catches the mistakes that no spell checker—or AI—can find.

The Takeaway for Developers

Eden's experiment is a powerful reminder that documentation is a user interface. It's the first thing many people see, and it can be the deciding factor between adoption and abandonment. The cost of poor documentation is measured in lost users, wasted time, and increased support requests.

The solution is not to spend more time staring at your own words, but to get them in front of real people. You don't have to pay them if you can't afford it, but finding a few individuals who will talk aloud while they try to follow your instructions is an investment that will pay dividends. The process is uncomfortable, but the result is a README that works for the people it's designed for.