Guide / Verification

How to verify AI-generated code before you trust it.

A checklist, and four levels of evidence.

Verify agent-written code the way you would verify a stranger's pull request: do not take the author's word that it works. Run the existing tests and type checks, have someone other than the author read the diff, reproduce any suspected problem as something executable, compare against the code before the change, and then decide yourself whether the behavior is what you intended.

The last step cannot be automated. An agent saying all tests pass tells you about the tests it chose to run. A tool reporting a failing test tells you an assertion fails. Only you know what the code is for.

A verification checklist

Run the project's own suite yourself. Not the agent's summary of it. Check that no test was deleted, skipped or loosened in the same change.
Run the type checker and linter. They are cheap and catch a class of errors models still make.
Read the diff with a second reader. A person, a fresh agent session, or a different model. The author's session shares the author's assumptions.
Turn each suspicion into something that runs. A test, a script, a command with its output. A suspicion you cannot reproduce stays a suspicion.
Run the reproduction on the code before the change. If it fails there too, the change did not cause it.
Decide against intent. Is the failing behavior wrong, or is the test's expectation wrong?
Note what nobody checked. Files no test touched are unknown, not fine.
Keep the tests that mattered. They guard the fix against the next change. See keeping regression tests from findings.

Four levels of evidence

These are easy to confuse, and each is weaker than it sounds until the next one is established.

Level	What it shows	What it does not show
1. A failing assertion	One run of one test did not get the value it expected	That it will fail again, or that the expectation is right
2. Reproducible behavior	The same failure on every run, in a clean environment	That the change caused it
3. A regression	The test passes before the change and fails after	That the old behavior was the intended one
4. A violation of intent	The behavior contradicts what the code is meant to do	Nothing further. This is a bug, and a person established it

Level 3 has traps of its own. A test can pass on the old code only because the feature did not exist there. A longer treatment, with examples from graded reviews, is in why a failing test is not sufficient evidence of a bug.

Doing it with arnx

arnx automates checklist steps 4 and 5 and prepares step 6 for you. It maps onto the levels like this:

Level 1 to 2: each generated test runs three times in Docker, on a throwaway copy with no network. A finding failed in every run.
Level 3: each failing test is rerun on the code before the change. The report labels the outcome: regression, pre-existing, new code, changed or unknown.
Toward level 4: a second model reads the test, the failure and the diff and confirms or disputes it. Then arnx triage asks you. Where intent is unclear, arnx asks a question, and your answer becomes a test.
Step 7: the review lists changed code no test reached and says partly covered or not tested.

After the agent's change

arnx init
arnx review
arnx triage
arnx accept
arnx check

Examples of each level being settled: a regression in beets that a person then confirmed against the real command, and a Hummingbot finding that was reproduced by hand on the project's current head before anything was sent upstream.

Asking the agent to verify its own work

You can ask a coding agent to do this for you, as long as the checker is not just the author rereading itself. With the arnx skill installed, the agent runs the review, reports the findings and stops for your marks. A prompt to copy is on the agent page. Guides for specific agents: Claude Code and Codex.

Limits and trade-offs

Verification by generated tests only covers behavior a test can reach. Design, readability and most security properties need a reader.
The second model is often wrong when it agrees. In our grading, 12 of 36 findings it confirmed were real defects, while 22 of 23 it disputed were rightly disputed. See the judge note.
arnx does not type check Python and does not run browser-only tests. Keep your own tools for those.
It costs model calls: each review stops writing tests at review_usd, $3.00 by default.
Finding nothing does not verify the change. It means the tests that were written passed.

To try the workflow on a change with a known answer, use the checkout demo. Install steps are in the quickstart, and what each label means is on the product page.

Last reviewed 2026-11-12 against arnx 0.1.11. The release documentation is canonical for your installed version.