Research

What holds up, and what does not.

Notes from grading arnx's own findings, with the counts and the limits.

We build arnx, so we have a stake in these numbers. That is why each note gives its method, its sample, its raw counts and what is missing for someone else to repeat it. The samples are small. None of these notes is a product-wide accuracy figure, and none should be quoted as one.

• AI code review findings on unfamiliar repositories

  65 graded findings from 18 reviews of 12 repositories new to arnx: 3 real bugs, 13 minor defects, 26 intended behavior, 12 wrong tests, 11 unreachable.

• Can a second LLM judge code review findings?

  In one batch, a second model's dispute was right 22 of 23 times; its confirmation held 12 of 36 times. Read the two verdicts differently.

• Why a failing test is not sufficient evidence of a bug

  Four levels of evidence, from a failing assertion to a violation of intended behavior, with counts from one graded batch and two public examples.

How to read these notes

• The grades are ours. Maintainers of the reviewed projects did not grade anything, except where a pull request was merged.
• All three notes draw on one batch of 18 reviews made with one pre-release build. A different build, model mix or set of commits can give different counts.
• The raw review files are not published yet. Each note says what a reader would need to reproduce it.

For individual findings you can check against a public pull request, read the case studies. For how arnx labels evidence in a review, read the product page.