Can You Compile Test Cases Into Automation Like Source Code?
Can you treat test cases like source code that compiles into automation?
I wondered if this might work with an AI prompting approach. In this 13-minute experiment, I'm exploring how far I can take building a fully prompt-driven test automation system using Claude Code and Chrome DevTools MCP. The core idea is treating markdown test cases as source code that gets compiled into executable YAML files through a 4-step process. Sort of similar to how a programming language compiler turns source code into machine code. I'm combining everything I learned from Experiments #2 and #3 into what I'm calling the Test Automation Compiler approach.
The compilation workflow follows four commands that transform human-readable test documentation into production-ready automation:
First, the /discover command executes the markdown test case while capturing everything needed for automation—the best selectors, timing requirements, screenshots, and patterns in the application behaviour.
Second, the /learn command analyses that discovery log and extracts automation intelligence specific to the application, identifying the most reliable selectors and optimal wait strategies.
Third, the /generate command takes those learnings and the original test case to create a YAML specification that can be executed consistently by an AI coding tool.
Finally, the /validate command runs that generated YAML to verify the compilation was successful. It also scores the automation run across multiple dimensions including reliability, timing, data handling, determinism, and maintainability.
The results from this first test case were surprisingly good. The validation run gave the compiled automation a 10/10 score across all five dimensions. Further testing is required but it kind of confirms that this automated test is production ready and fully deployable to CI/CD.
The key concept in all of this is that the markdown test case remains the single source of truth. The YAML specification is automatically compiled from actual observed behaviour rather than assumed implementation. No code was written manually. The entire workflow is prompt-driven using four clear commands.
My initial assessment is that this approach is taking shape but still needs a degree of validation. The 4-step workflow feels natural and systematic, and achieving 10/10 reliability on the first attempt with no tweaking is encouraging (although this wasn't the toughest of test cases to challenge the process with). The concept of treating test cases as compilable source code works both as a mental model and seems to work in practice.
However, I'm very aware this is currently Chrome-only because we're using teh Chrome DevTools MCP. This needs extensive testing with more complex scenarios too. The feedback loop isn't implemented yet, and I haven't validated reliability in production GitHub workflows. I'm curious whether this will work reliably in CI/CD and whether the Playwright MCP could enable multi-browser support. The foundation feels solid enough to invest more time though. So more work to prove it scales beyond this single test case.
If you're interested in trying this yourself, I'll be releasing everything on GitHub once the feedback loop is complete. All four slash command implementations, the complete Test Automation Compiler strategy document, example test cases with expected outputs, and (possibly) CI/CD integration examples. You can read the full experiment writeup at https://www.testmanagement.com/blog/2...
Subscribe to see how this develops in my weekly AI testing experiments: / @testmanagement and sign up for prompts and learnings at https://www.testmanagement.com/ai