Guide / AI code review

AI code review: comments versus executable evidence.

Two kinds of output, and what each one costs you.

AI code review means a language model reads a code change and reports what looks wrong. Most tools report in prose: a comment on a line, often with a suggested fix. A smaller group reports something you can run: a test that fails on the change. Both are useful. They fail in different ways, and they fit different points in the workflow.

The short version: use comment-based review for breadth on every pull request, and executable evidence where a wrong claim is expensive to chase or a real bug is expensive to miss. Neither replaces your own tests or a person who knows what the code is meant to do.

What a review comment gives you

A comment is a claim about the code. It is cheap to produce and quick to read. It can cover style, naming, security patterns, missing error handling and design, including things no test can express. Its weakness is that you must verify it yourself. A model can describe a bug that is not there in the same confident tone as one that is. When that happens often, people stop reading the comments.

What executable evidence gives you

A generated test turns the claim into inputs, an expected result and an observed failure. You can rerun it, read it and argue with it line by line. If it also passes on the code before the change, you know the change altered that behavior.

A failing test is still not proof of a bug. The test can encode the wrong expectation, or reach a state the real program never reaches. In our own grading of 65 findings on repositories we did not know, more findings tested intended behavior than tested real defects. See why a failing test is not sufficient evidence of a bug and the full counts.

Executable evidence also costs more. Something must build the project, run the tests in isolation and pay for the model calls that wrote them. It only works where the project has a test runner the tool supports.

Where each kind fits

A rough guide, not a ranking
Situation	Comments	Executable evidence
Every pull request, whole team	Good fit: fast, no setup per change	Possible, but slower and priced per run
An agent just finished a change	Useful as a second read	Good fit: checks behavior before a person reads the diff
Style, design, security patterns	Good fit	Poor fit: not expressible as a failing test
Logic, edge cases, regressions	Can point at them	Good fit: reproduces them
Guarding the fix later	Nothing to keep	The test stays as a regression check

The two combine well. Run evidence-based review before you open the pull request, and let a comment-based reviewer read the pull request. See how arnx sits beside CodeRabbit, Greptile and Claude Code's own review.

Questions to ask of any AI review tool

What does a finding consist of? Prose, a suggested patch, or something you can run.
Does it run your code? If so, where, with what network access, and with which secrets in reach.
What leaves your machine? The diff only, or files around it, and to which model providers.
How does it handle being wrong? Look for a way to dismiss or dispute a finding, and for published numbers on how often findings hold.
What does silence mean? A tool that reports nothing should say what it did not look at.
What does it cost per change? Per seat, per review or per token, and whether there is a cap.
What do you keep? A resolved thread, a rule, or a test.

A workflow to try

This is the evidence-based loop with arnx, an open-source CLI. You need Docker and a model key, or arnx Cloud.

After a code change

npm install -g @arnx-dev/arnx
arnx init
arnx review
arnx triage
arnx accept
arnx prompt
arnx check

arnx review writes tests for the change, runs each three times in Docker, reruns the failures on the code before the change and asks a second model to confirm or dispute each one.
arnx triage is where you mark each issue real, intended, noise or rewrite. This step is the review. The rest is preparation.
arnx accept keeps the tests of real issues. arnx prompt hands the fix to your coding agent. arnx check reruns the kept tests with no model calls.

Real examples: a shell completion crash in beets, valid SQL rejected by SQLFluff's parser, and a path escape in arnx's own remove command.

Limits and trade-offs

A review takes minutes and model spend. Each arnx review stops writing tests at a cap, review_usd, $3.00 by default. Our own reviews ranged from $0.17 to $3.19.
A second model's agreement is a judgement, not a proof. See can a second LLM judge code review findings?
Model replies vary. Two reviews of one change can report different findings, and finding nothing does not show the change is correct.
arnx covers Python, JavaScript, TypeScript and Rust (early), on changes up to 200 KB, and does not run tests that need a browser.

To try it: the quickstart, or the checkout demo repository, which has one planted bug to find. What a finding and a check mean is on the product page.

Last reviewed 2026-11-12 against arnx 0.1.11. The release documentation is canonical for your installed version.