Guide / Regression tests

Keeping regression tests from AI review findings.

A finding you fixed should stay fixed.

When an AI review produces a test that exposes a real bug, keep the test. It already fails on the bug and passes on the fix, which is what a regression test is. Before you keep it, read it, make sure it tests behavior you intend, and put it somewhere it cannot be quietly edited until it passes.

That last point matters most with coding agents. An agent asked to make a failing test pass has two ways to do it: fix the code or change the test. A test changed until it passes shows nothing.

Is this test worth keeping?

A model wrote it. Read it as you would read a stranger's code.

• Does it assert intended behavior? If the expectation is wrong, the test will hold a bug in place.
• Does it reach the code through a real path? A test that calls a private helper with inputs the program never produces guards nothing.
• Is it deterministic? No clock, network, ordering or shared state it does not control.
• Does it check behavior, not wording? An assertion on an exact error string breaks on the next rewording.
• Is it the smallest test for this bug? One cause, one test, is easier to trust later.

If the bug is real but the test is wrong, say so and have it rewritten. Do not keep it as it is.

Where a retained test should live

There are two reasonable homes. Your own test folders, where your runner collects it with everything else. Or a separate, pinned place that only a dedicated command runs. arnx uses the second:

• Accepted tests are kept in .arnx/tests/. Your project's own test run never collects them.
• Each file's sha256 is pinned in .arnx/accepted-tests.json.
• Only arnx check runs them, in Docker, in a throwaway copy of the repository with the tests put in place.
• arnx never edits your code or your tests.

The trade-off: your suite's layout and conventions stay untouched and model-written code is clearly marked as such, but you have a second command to run. If you prefer the test in your own suite, copy it there by hand after review. It is then yours to maintain.

From finding to kept test

arnx triage
arnx accept
arnx check
arnx prompt
# fix the code under test, never the tests
arnx check
git add .arnx/tests .arnx/accepted-tests.json

1. Triage. Mark each issue real, intended, noise or rewrite. Issues backed by a confirmed test come marked real. Change what you disagree with. Rewrite means the bug is real and the test is wrong; add what it should check.
2. Accept. Tests of issues marked real are kept. Tests of everything else are left out. Your answers to arnx's questions can become tests too; arnx accept --no-answers skips that model call.
3. Check, and expect red. The tests of the bugs you just accepted fail. That is the list to fix.
4. Fix and check again. arnx prompt gives your coding agent the failing tests and your answers. Green means the accepted tests pass now.
5. Commit the tests with the fix. .arnx/tests/, .arnx/accepted-tests.json, and .arnx/answer-watch.json when there is one.

From then on, arnx check on a later change reruns every kept test with no model calls. One such test still guards arnx's own code: see the path escape in arnx remove. In the beets and Hummingbot cases, the test went upstream inside the pull request, in the project's own suite.

When a kept test is edited

If an accepted test file, or its entry in .arnx/accepted-tests.json, changes after accept, arnx check runs nothing and exits 4. You then choose:

• arnx accept --again <id> restores the review's tests and overwrites the edit.
• arnx accept --keep-edits records the current files, after you have read and agreed with the change.

Both are the developer's to run. The arnx skill tells coding agents never to touch accepted tests or run arnx accept.

Your own suite still matters

A passing arnx check covers only the accepted tests. Run each touched project's full suite after the fix as well. Kept tests are a net under specific bugs you have seen. They are not coverage.

Limits and trade-offs

• Kept tests are model-written code in your repository. Budget time to read them. Unread, they are a liability.
• A kept test with a wrong expectation blocks correct changes later. In our grading, many confirmed findings tested intended behavior, so this is a real risk: the counts.
• arnx check needs Docker and the environment in .arnx/review.yaml. It is not as light as your unit tests.
• Tests outside your suite do not show up in your coverage or your runner's reports.
• Rust is early: generated tests are integration tests that see the crate's public API only.

To see a test go from finding to kept check, run the checkout demo through arnx accept and arnx check. The reference is keep the useful tests in the quickstart.

Last reviewed 2026-11-12 against arnx 0.1.11. The release documentation is canonical for your installed version.