Research / Field study

AI code review findings on unfamiliar repositories.

65 failing tests, graded one by one. About one in four was a real defect.

We pointed arnx at open-source repositories it had never been run on, one commit each to start, and graded every finding it reported. Of 65 findings, 3 were real bugs and 13 were real but minor defects. The other 49 were not defects: 26 tested behavior the project intends, 12 came from a wrong test, and 11 described a scenario the real program cannot reach.

One batch, one pre-release build, graded by us. These counts describe this sample. They are not an accuracy rate for arnx, and later releases changed behavior this batch exposed.

Method

• Build. A local pre-release build of arnx 0.1.8. No public release matches it exactly.
• Repositories. 16 TypeScript and Python open-source repositories that arnx had not been run on before.
• Commits. Chosen by a rule fixed before any run: the largest source change of 40 to 400 lines not made by a bot, dated 28 September to 12 October 2026, and not documentation or dependencies. One exception: the top pick in one repository was a translation bot's commit, so the next one ran. A second commit per repository ran until a spending limit stopped new runs, so some repositories have two reviews and some have one.
• No help. Each run was a first arnx review of that commit, with no hints, no hand-written configuration and no retries to get a better result. One review was rerun after a provider outage cut the first attempt.
• Limits. $1.50 per review and $0.75 per setup. The first repository ran under lower limits.
• Models. Reviewer: claude-sonnet-5-5 in every review. Second opinion: GLM-5.3-Flash through River, with gpt-6.1-sol as the fallback when it timed out. The cost line of every review also shows gpt-6.1-sol, the default harness's second hunter.
• Grading. Two graders read the code behind each finding and reran the ones judged true in Docker with no network. The graders were AI agents working under the direction of arnx's author. No maintainer graded anything.

Sample

Of the 16 repositories, 12 produced at least one complete review. Those 12, with 18 complete reviews between them, are the sample. All are public open-source projects in Python or TypeScript. Two of them, SQLFluff and beets, have findings that were sent upstream and are written up in the case studies.

The other 4 produced no complete review:

• Two: the first review could not set the project up within its 10-minute limit.
• One: setup found only a web UI there, which arnx does not set up.
• One: arnx refused the change because a test file held text that looked like a private key.

In all, 28 runs were started: 18 complete, 3 refused, 2 stopped at the setup limit, 2 stopped at setup for a web UI, 2 incomplete and 1 stopped before review. We do not list the other repositories here: some findings graded real have not been reported to their maintainers yet, and they should hear it first.

Raw counts

The 18 complete reviews wrote 374 tests and reported 65 findings. A finding is a generated test that failed.

Of the 16 real findings, 15 were still present at the project's head when graded. One real bug had already been fixed upstream.

The second table is the subject of Can a second LLM judge code review findings?

One repository holds 18 of the 65 findings. A sample this small leans on whichever commits happened to be picked.

Classification criteria

• Real bug. The program does something wrong that a user can hit, and the test shows it.
• Real, minor. A true defect with small effect: an edge case, a rarely used form, or a gap in new support.
• Intended behavior. The test fails, but the code does what the project means it to do. The test's expectation was the model's guess.
• Wrong test. The test itself is faulty: an invalid input it calls valid, a broken fixture, or an assertion that does not check what it claims.
• Unreachable scenario. The failure is real inside the function under test, but no caller in the program can produce that input or state.

Cost and time

The 18 complete reviews cost $7.42 in model spend and took 111 minutes in total. One review ranged from $0.07 to $0.96 and from 79 seconds to 13 minutes 39 seconds. Across all 28 runs, reviews cost $7.54 and setups $1.05. These are observations from this batch, not prices.

Limitations

• The grades are an estimate by us and by AI graders we directed, not the maintainers' word. A grade of intended or unreachable can be wrong.
• One pre-release build. Releases since then changed how findings are labelled and when a run fails.
• 65 findings from 18 reviews is a small sample, and one repository supplies more than a quarter of it.
• TypeScript and Python only. No Rust, no other language.
• Model replies vary. A second review of the same commit can report different findings.
• We counted what arnx reported. We did not count the bugs it missed, so this says nothing about recall.
• Commits of 40 to 400 lines from a two-week window are not a random sample of all changes.

Conflict of interest

We build arnx and sell arnx Cloud. We chose the repositories, ran the reviews and did the grading. Nobody outside checked this work.

Artifacts

The raw review files (each review's report, generated tests, test output and second-opinion record), the grade sheet with its evidence notes and the run ledger are kept by us and are not published yet. To reproduce this study a reader would need the list of commits, the exact build, and those files. Two findings can be checked today against public pull requests: SQLFluff and beets.

Related: Why a failing test is not sufficient evidence of a bug.

Last reviewed 2026-11-12 against arnx 0.1.11. The release documentation is canonical for your installed version.