Research / Evidence
Why a failing test is not sufficient evidence of a bug.
A test can fail honestly and still be wrong about what should happen.
A failing test shows that one assertion does not hold for one input. It does not show that the code is wrong. Whoever wrote the test chose the input and the expected result, and when a model wrote it, both are guesses about what the project intends. In one batch we graded, 49 of 65 failing generated tests were not defects.
Four levels of evidence
Each level adds something the one before it lacks. Only the last one is a bug.
- A failing assertion. The test ran and an assertion failed, once. This rules out very little. The test may be flaky, the fixture may be broken, or the input may be invalid.
- A reproducible behavior. The test fails the same way every time. Now you know what the code does with that input. You still do not know whether that is wrong. arnx runs each test 3 times for this reason, and a test that fails only sometimes is reported as flaky.
- A regression caused by the change. The same test passes on the code before the change and fails after it. This ties the behavior to the change. It still assumes the old behavior was the intended one, and that the pass on the old code was a real pass.
- A violation of intended behavior. The code does something the project does not mean it to do, in a situation real use can reach. This needs a fact from outside the test: a specification, an upstream reference, or the owner's word.
A tool can establish the first three by running code. The fourth takes knowledge of intent. That is why arnx asks the developer to mark each issue real, intended, noise or rewrite, and keeps only tests of issues marked real.
Where failing tests went wrong in one batch
The sample is the one in AI code review findings on unfamiliar repositories: 65 findings from 18 reviews of 12 open-source repositories, graded by us. Every one of the 65 was a generated test that failed in every run. So all of them had reached the second level.
| Grade | Findings | What the failing test got wrong |
|---|---|---|
| Intended behavior | 26 | The expected result. The code does what the project means. |
| Wrong test | 12 | The test itself: an invalid input, a broken fixture, or an assertion that checks something else. |
| Unreachable scenario | 11 | The input. No caller in the real program can produce it. |
| Real, minor | 13 | Nothing. A true defect with small effect. |
| Real bug | 3 | Nothing. |
The largest group, 26 of 65, failed on intent alone. The test was sound, the failure repeated, and the code was right. No amount of rerunning separates these from real bugs. The 11 unreachable scenarios are similar: a unit test calls a function directly with a state the rest of the program never builds.
These counts describe one small batch on one pre-release build. They are an illustration, not a rate.
Two public examples
A confirmed test that asserted invalid SQL should parse
In a review of SQLFluff's ClickHouse projection grammar, arnx wrote a test
asserting that a projection with both GROUP BY and
ORDER BY should parse. The test failed in every run. The second
model confirmed it. It was wrong: ClickHouse itself rejects that form, so the
grammar was right to refuse it. The failure was reproducible and the test was
still mistaken about what is valid. Settling it took reading ClickHouse's own
source. Details:
the SQLFluff case study.
A real gap filed under already failing before
The same review held a test showing that a projection with a bare
SELECT was unparsable. That one was a real gap, and the fix was
merged upstream. But the test also failed on the code before the change,
because the parent commit had no projection support at all. arnx listed it
under already failing before this change
, the section for failures the
change did not cause. The label followed the rule and still pointed away from
the one true finding. A before-and-after comparison answers
did this change break it
. It does not answer is it broken
.
The third level can mislead in the other direction too. A test can pass on the old code only because the feature did not exist there, and then look like a regression when the new code handles the input differently. In the batch above we saw that in several findings. See Can a second LLM judge code review findings?
Questions to ask of any failing generated test
- Is the input valid? Check it against the specification or the upstream system, not against the test's name.
- Can real code reach this? Find a caller that can produce the input or state. If the test builds the state by hand, ask who else could.
- Who says the expected result is right? If the only source is the model that wrote the test, treat it as a question for the owner.
- What did the old code do, and why? A pass or a failure before the change means little when the feature is new.
- Does it fail for the stated reason? Read the failure message. A setup error or a missing import can look like a failed assertion in a summary.
What a failing test is still good for
A review comment says something might be wrong. A failing test gives you an input, an expected result and an actual result you can run yourself. Being wrong in a checkable way is cheaper to settle than being vague. In the batch above, each of the 49 non-defects could be dismissed by answering one of the questions above. The 16 real ones came with a reproduction attached. The test is where the judgment starts, not where it ends.
Limitations
- The counts come from 65 findings in 18 reviews, graded by us and by AI graders we directed, not by the projects' maintainers.
- One pre-release build of arnx 0.1.8, TypeScript and Python only. Later releases changed how before-and-after outcomes are labelled.
- We build arnx. Nobody outside checked the grading.
- The raw review files and the grade sheet are not published yet. The two examples above can be checked against the public SQLFluff pull request.
Last reviewed 2026-11-12 against arnx 0.1.11. The release documentation is canonical for your installed version.