Passante A Passante publication

En Passante

In Passing
Field Kit

The test was written after the fix

A fix is unproven until a test that failed on the broken code passes on the fixed code.

What happened

On 2026-09-22, a review cycle closed edge-case defects, a recursion over sensitive field names and an identifier mapping error, but each regression test was written after the repair, so no test had ever run red against the broken code and no fix was proven.

2026-09-22 any language · a test suite under review measured: none recorded
3 prompts, in order
  1. Reproduce the defect in one test A defect is reported and the code is still untouched.
  2. Prove red turns green A failing test exists and the fix is ready to apply.
  3. Back-fill tests for landed fixes Review findings were fixed already and no tests came with them.

1Reproduce the defect in one test

Run it when
A defect is reported and the code is still untouched.
It asks you for
The exact symptom, the test command, and where similar tests live.
You get back
One test plus the verbatim output of its failing run.
It refuses if
The test passes on unmodified code, so the defect is not reproduced.
copy this into a model
<stance>
A defect report is not a fix. Write the test first, and make it fail for the reported reason before any production code moves. A test authored after the repair goes green on its first run and says nothing about the defect. Ordering is the whole method here.
</stance>

<inputs>

1. The exact reported symptom. Ask for the input, the expected output and the actual output.
2. The command that runs the test suite.
3. The file or directory where similar tests live.
4. Confirmation that the fix is not applied yet.
Stop and ask if any answer is missing. Do not guess the test command.

</inputs>

<method>

1. Restate the defect as one observable claim. Name the input and the wrong output.
2. If the symptom is vague, ask for one concrete failing input. Do not invent one.
3. Find the smallest unit that produces the wrong output. Prefer a unit test over an end to end test.
4. Write one test that asserts the correct behaviour. Cover only this symptom.
5. Run the test command. Paste the real output.
6. Read the failure message. Confirm it names the defect's symptom.
7. A failure caused by a missing import or a typo in the test is not reproduction. Repair the test and run again.
8. If the test passes, report that the defect is not reproduced. Stop there.
9. Record the failing output verbatim. It becomes the baseline for the fix.

</method>

<deliver>

- The defect as one observable claim
- The test, in full
- The exact command used
- The verbatim failing output
- Whether the failure matches the reported symptom

</deliver>

<refuse>

- Do not change production code in this step. Not one line.
- Do not call a test failing without pasting the run output.
- Do not treat a broken test setup as evidence of the defect.
- If the test passes on unmodified code, stop. Say the defect is not reproduced and ask for a different input.
- If the test command is unknown, stop and ask for it.
- Do not write more than one test here. Extra cases hide which assertion carries the proof.

</refuse>

2Prove red turns green

Run it when
A failing test exists and the fix is ready to apply.
It asks you for
The failing output, the command that produced it, and the proposed change.
You get back
Three runs, before, after and fix-removed, with a verdict.
It refuses if
No captured before-run output exists for the target test.
copy this into a model
<stance>
The suite's result is the evidence, and it counts only in sequence. Red first, then green, with the same test and the same command. A test that already passes before the change is measuring something other than this defect. Both runs get recorded.
</stance>

<inputs>

1. The failing test and the verbatim output of its failing run.
2. The command used for that run.
3. The proposed change, as a diff or a described edit.
Wait for all three. Do not reconstruct a missing diff from memory.

</inputs>

<method>

1. Read the before-run output. Confirm it fails on the defect's symptom.
2. If no before-run output exists, stop. Send the reader back to produce one.
3. Apply the change exactly as given. Change nothing else.
4. Run the same command. Paste the output.
5. Compare the two runs. The target test must move from fail to pass.
6. Run the full suite once. Note any test that newly fails.
7. A newly failing test means the fix is incomplete. Report it.
8. Now remove the change and keep the test. Run the command again.
9. The test must fail again. If it passes, the test is not tied to this fix.
10. Restore the change and report all three runs.

</method>

<deliver>

- Before output
- After output
- Fix-removed output
- Verdict: proven, or not proven
- Any newly failing tests

</deliver>

<refuse>

- Do not edit the test while the fix is applied. That hides which side changed.
- Do not call the fix proven on one green run.
- Do not summarise output you did not capture.
- If the test passes with the change removed, stop. Report that the test does not cover the defect.
- If the suite cannot run, stop and say so.
- Do not widen the fix to make an unrelated failing test pass.

</refuse>

3Back-fill tests for landed fixes

Run it when
Review findings were fixed already and no tests came with them.
It asks you for
The finding list, the change for each one, and the test command.
You get back
One test per finding plus its fix-removed run result.
It refuses if
A finding's change cannot be removed on its own.
copy this into a model
<stance>
Fixes that land during review often arrive with no test behind them. Each one needs a test that can still see the original defect. Remove the fix and the test must go red. An assertion that holds either way locks in nothing.
</stance>

<inputs>

1. The list of findings, one line each.
2. The code change that addresses each finding, by file and region.
3. The command that runs the test suite.
4. Whether each change can be reverted on its own.
Wait for the list before writing anything. Do not infer findings from the diff alone.

</inputs>

<method>

1. For each finding, restate the defect it fixed as one observable claim.
2. Write one test per finding. Name the test after the defect, not after the fix.
3. Run the suite with all fixes in place. Every new test must pass.
4. If a new test fails here, the fix or the test is wrong. Report it and move on.
5. Now remove one finding's change. Leave the others alone.
6. Run the command. The matching test must fail.
7. Restore the change and run again. The test must pass.
8. Repeat steps five to seven for each finding, one at a time.
9. If a change is interleaved and cannot be removed alone, mark that finding unproven.
10. Report every finding with no test that goes red.

</method>

<deliver>

- Finding list with status, proven or unproven
- The tests, in full
- The command used for each run
- Fix-removed output per finding
- Findings still lacking a test that can detect the defect

</deliver>

<refuse>

- Do not mark a finding covered without a red run against the fix removed.
- Do not bundle several findings into one test. The proof stops being readable.
- Do not remove more than one change at a time.
- If no change can be isolated, stop. Report that none of the fixes are proven.
- Do not describe a run you did not perform.
- Do not treat the reviewer's approval as evidence that the defect is gone.

</refuse>
How you know it worked

Remove the fix and keep the test. Run the test command. The test must fail and name the defect's symptom.

from a real incident, 2026-09-22 · stack any language · a test suite under review · verified by none recorded · testingregressiontddcode-review

← every Field Kit