Blind testbench pattern

One thing that makes it hard to learn is the curse of knowledge. You know what you want to create, so when you look at what you made your mind fills in the gaps, making it impossible to evaluate what you’ve created honestly. To learn what you really need is to experience a creation like a reader or user coming blind to the experience.

I may have stumbled into a fantastic learning tool, the blind testbench. Essentially, with this pattern I use AI to generate a stock of candidate content that gets saved to a file. Then I, as the user, use an app to try out the candidates blind, recording my thoughts.

The power of this process is getting to experience the content without knowing exactly what it’s meant to do or mean. I encounter it as a new user would, not as a content creator who is cursed with the knowledge of my own intent.

The pattern
I’ve started using this for my writing exercises. Using a Claude skill, I stock a candidate pool with writing exercises. Each is its own file. I then attempt the writing exercises Claude generates, recording a grade (for the exercise, not my writing), and jotting down my thoughts after reading Claude’s rationale for the exercise. The point isn’t to use the exercises later. There’s no pipeline connecting it to my editorial dashboard. The point is to learn what makes a good writing exercise so I can write better exercises myself. It’s a bit like coffee cupping, or a drinking a flight of wine to train taste.

There are four main parts to this system.

1. File Contract, a specification for candidate files. The form of the contract depends on what you are studying.
2. Stocking Skill, a skill for an AI agent to run to research and generate candidate files. Importantly, the skill starts by looking at what exists in the stock and planning to explore new territory.
3. Evaluation Bench, an simple app to test, grade, and review each candidate, appending feedback to the file. It can also record user interactions like time on task, time away, and word counts.
4. Learning Ledger, a file that collects insights from the testbench results. While the user learns from the process directly, the system learns indirectly by running through the feedback and adding insights to a ledger.

Examples and build instructions
Already I have four examples I’m using or building.

• Writing Gym: Writing exercises for my daily writing prompts
• Cookie Jar: Data for a number-estimation game
• Kotoba Reader: Short Japanese texts on culture with a vocabulary quiz
• Rhetorical Reader: AI-generated passages shaped by rhetorical situation (in progress)

If you’re interested in making a blind testbench, here’s are links to PATTERN - Blind Testbench and PATTERN - File First Collaboration that can be fed to an LLM to set up a similar testbench.