Case study / arnx

arnx remove could delete a file outside the repository.

arnx reviewing its own code, before the first release.

Before 0.1, arnx reviewed the change that added its own arnx remove command. It wrote a test showing that a path recorded in .arnx/accepted-tests.json could point outside the repository, and that remove would delete the file there. The test, the failure, the fix and the retained regression test are all in the public source.

Conflict of interest: this is arnx's authors using arnx on arnx. We chose the example and we judged the finding. Read the SQLFluff and beets pages for code we did not write.

1. Repository and reviewed commit

arnx-dev/arnx, review 20261022-227eb7, on 2026-10-22, of the change that added arnx remove. The command lists what arnx put in a repository and, with --yes, deletes it. With --tests it also deletes accepted tests that an older arnx kept in the project's own test folders.

2. Expected behavior

arnx remove deletes only files inside the repository it runs in, whatever .arnx/accepted-tests.json says. That file is committed and shared, so its contents can come from anyone with write access to the repository.

3. Actual behavior

With a record that listed ../victim.txt, arnx remove --yes --tests deleted victim.txt in the parent directory.

4. Suspected defect

arnx suspected that remove trusted the paths recorded in .arnx/accepted-tests.json: it read the keys as written and joined each one to the repository path before deleting.

5. Generated test

Written by arnx, quoted from the README

test("S2 accepted-test paths escaping the repository are never deleted", async () => {
  const { base, repo, env } = setup();
  const victim = join(base, "victim.txt");
  put(victim, "keep me");
  put(join(repo, ".arnx", "accepted-tests.json"), record({ "../victim.txt": {} }));
  const cli = new Cli({ cwd: repo, env });
  await withDocker("/nonexistent/docker", () => cli.run("remove", "--yes", "--tests"));
  expect(existsSync(victim)).toBe(true);
});

6. Execution result

From the review's report, quoted from the README

### Issue 1 · remove --yes can delete files outside the repository
- Actual: In S2, `../victim.txt` outside the repository was deleted. …
- Failing tests: F2, F4

### F2 · regression · confirmed by the second opinion
- Failure: `expect(received).toBe(expected)`
- Before the change (regression): passes on the code before the change (3 runs), fails with it
- Second opinion: planRemove takes accepted-tests.json keys verbatim and applyRemove rmSync's join(repo, p), …

The test failed in all 3 runs and passed on the code before the change. The second model agreed with the explanation. The excerpt is shortened in the README, and so here. We do not publish the cost or duration of this review.

7. Correction

We marked the issue real and arnx accept kept the test. A coding agent fixed remove from arnx's fix prompt. In the released source, the fix is in two places in src/remove.ts:

When the accepted-tests record is read, paths that would leave the repository are dropped. The plan of what to delete never contains them.
The step that deletes checks each path again and skips any that is not inside the repository.

8. Verification result

arnx check showed the accepted test passing after the fix. The test is still in the public repository, as accepted, in tests/remove-safety.test.ts, next to the other tests arnx wrote in the same review. It runs with the rest of arnx's suite.

9. Upstream references

README: What a finding looks like, with the test and report excerpt.
tests/remove-safety.test.ts, the retained regression tests.
src/remove.ts, the corrected code.

10. Limitations and observations

The defect never shipped. The review ran before the first public release, on a change that was still in development. No released version of arnx had it.
We picked this example because it is a clean one. The README says so too: in our own reviews, some confirmed issues tested behavior we intended. Deciding is the developer's part.
The review record itself is not published, only the excerpts in the README. The test and the fixed code are public and can be read and run.
Reaching the defect needed a crafted record file in the repository. That is a realistic input for a file teams commit, and still a narrower case than a crash on ordinary use.

Last reviewed 2026-11-12 against arnx 0.1.11. The release documentation is canonical for your installed version.