Achieving 12x faster test authoring with AI QA agents

TL;DR

Writing, running and reporting tests took close to an hour per ticket before a single test was executed.

QA flow is an AI QA agent that reads any spec, writes the test cases, runs them unattended and files the bugs it finds.

Authoring dropped from about an hour to five minutes, returning about 22 hours a week per engineer.

Every ticket starts with a specification and someone reading it: a user story, a design frame, a pull request. On anything non-trivial that is close to an hour of work before a single test runs, and several people digest the same document separately and reach slightly different answers.

QA flow does that first pass. It reads whatever spec exists, writes the cases a QA engineer would write, runs them unattended and files what breaks. Five minutes instead of an hour, with one engineer reviewing a complete draft rather than a blank page.

The challenges

Every ticket runs the same loop before anything gets tested. It is careful, necessary work, and it takes most of the week.

Authoring from a blank page
Read the story, work out what the change touches, think through edge cases, write the cases out. Close to an hour before anything runs.

Execution with someone present throughout
Open the case, read step one, go do it, come back, read step two. Roughly eight minutes per test case, with attention on it the whole time.

Writing the failure up a second time
Anything that fails becomes a bug report, a second pass over work already done once.

The judgment goes last
Edge cases and impact analysis are what a QA engineer is for, and they get squeezed because authoring and execution fill the day first.

What we built

Three things stay with the engineer: approving the suite, adding the edge cases only they would think of, and fixing the bug. Everything else runs without anyone watching. It was built in-house by a four-person pod working full time for six months across the front end, the back end and the agent itself.

Conclusion

Authoring drops from about an hour per feature to five minutes. Execution drops from roughly eight minutes per test case with an engineer watching to a three-minute run that needs nobody, and the separate write-up disappears. What comes back is about twenty-two hours a week per engineer.

Want this pipeline running on your backlog? Get in touch.

Spec sources: Jira, Linear, GitHub repositories, screenshots
Test types: end-to-end, integration, unit, smoke, accessibility
Agent: fine-tuned model, false positive and negative rates tracked run by run across six months
Output: generated test cases, unattended execution, automatic bug tickets, results dashboard
Build: four-person pod, full time, six months across front end, back end and agent

FAQ