Picture a test lead whose dashboard has been solid green for weeks, and who reports it the way you'd report a fever that won't break. Her developers use an AI assistant to write the code. Her testers, under the same schedule pressure, use the same assistant to write the test cases. Every run passes. She has no idea whether the system works, and she is right to be uneasy.
The reason for her unease is older than any of the tools. In the Army we don't let the person who built the thing certify the thing. The unit that prepares a range doesn't score its own qualification, and the maintainer who turned the wrench is not the one who signs the inspection. This isn't distrust of the individual. It's an acknowledgment that a person checking their own work checks it with the same assumptions that produced it. Independence is the whole point of the check.
AI has quietly dissolved that independence on a lot of programs, and nobody decided to do it. It happened because the same assistant is open in every window. It drafts the requirement, generates the build, writes the test cases, summarizes the results, and drafts the go-live recommendation. Each step looks like a separate control. In practice they are one mind reviewing itself five times.
Shared blind spots look like agreement
When two independent people check a piece of work and agree, the agreement carries information, because they came at it from different directions and different mistakes. When the same model writes the code and the tests, the agreement carries almost none. If the model misread the requirement, it misread it in the code and then wrote tests that confirm the misreading. The test passes because it was derived from the same wrong idea of what "correct" means.
1. The tests mirror the implementation
An assistant asked to test code it just wrote tends to test what the code does rather than what the business needs it to do. Edge cases the author never considered stay unconsidered by the tester, because the author and the tester are the same pattern of reasoning. You get high coverage numbers and a suite that cannot fail in the places that matter.
2. The summary inherits the same gloss
The results narrative gets drafted by the tool that generated the evidence. A failed case is described as a known limitation. A skipped scenario becomes "out of scope for this cycle." Each sentence is plausible and none is a lie, and the reader ends up with a smoother picture than the data supports. I wrote about the same dynamic in the confidence problem: the prose sounds identical whether the underlying work is solid or hollow.
3. The human reviewer gets less than they think
The final reviewer sees a clean package. They check it for coherence, because coherence is what they can see, and a coherent package is exactly what a single model produces. This is how a review turns into a rubber stamp without anyone's attention flagging. The reviewer is not lazy. The package is simply built to be agreed with.
A check only counts if it could have come out differently. If the same source produced the work and the evidence, you have a single opinion repeated, not independent verification.
Put the separation back on purpose
None of this means keeping AI away from testing. The assistant is excellent at generating cases, finding combinations nobody would have tried, and grinding through regression volume at a speed no team matches. The problem isn't the tool. It's the structure around it. On programs I run, the fix is to restore the separation of duties that the tools erased, and to do it as an explicit design decision rather than a hope.
Concretely, the acceptance criteria are written by a human who owns the business outcome, before anyone generates a line of build or test, and they are written in terms of behavior the business can observe, not in terms of the implementation. The test design is derived from those criteria and not from the code. Where the same assistant has touched the build, a different tool or a different person does the verification, and the instructions they receive come from the criteria rather than from the build's own description of itself. A named human then reads a sample of the raw results, the failures and the skips and not the summary, and signs for what they read. The sign-off names the person. It does not name the tool.
None of this is expensive compared with what it protects. It's a few hours of senior attention placed at exactly the seam where the model can no longer catch its own mistakes. What it costs is the temptation to treat the speed as the saving. The speed is real, but the saving only counts if the checks still mean something at the end.
- Write acceptance criteria before any build or test is generated, and have a human who owns the business outcome write them in observable behavior.
- Derive test design from the criteria, never from the code. A test that describes what the code does can't tell you whether the code is right.
- Don't let one assistant author both the work and the evidence. Use a different tool, a different prompt lineage, or a different person for verification.
- Have a named human read the raw failures and skips, not the summary, and sign for what they actually read.
- Treat a long run of green results as a question to answer, not a result to report. Ask what would have turned it red.
The bottom line
The fix for that test lead isn't exotic. Her analysts rewrite the acceptance criteria in plain observable terms, a separate toolchain generates cases from them, and the pass rate drops. That drop is the best news the program will get all quarter, because every point that vanished was a defect that would otherwise have found the customer. A green board earns trust only when it was possible for it to be red. Build the independence in, name the person who signs, and the checks will mean something on the day you need them.