Passante A Passante publication

En Passante

In Passing
Field Kit

Fifty passing tests cannot clear 4,207

A subset you chose because it was broken can only report on itself, so no claim about suite health stands until the full run's exit status is read.

What happened

On 2026-09-24, a 50-test subset came back green and the suite was about to be called healthy on that basis, until the complete run was executed and reported 4,207 passes.

2026-09-24 a CLI test runner · 4,207-test suite measured: 50-test subset pass versus 4,207 full-suite passes
3 prompts, in order
  1. Scope what the subset proves Your targeted test run passes and you want to report progress.
  2. Run the full suite, read exit status The targeted tests pass and you now need a whole-suite result.
  3. Separate fixed tests from new breakage The full run shows failures and you need to know which ones you caused.

1Scope what the subset proves

Run it when
Your targeted test run passes and you want to report progress.
It asks you for
The exact selection you ran, the counts it printed, and the claim you intend to make.
You get back
A verdict on whether the subset supports your claim, plus the wording your evidence does support.
It refuses if
You cannot state how many tests ran in the targeted selection.
copy this into a model
<stance>
A green subset proves one thing. The tests you selected pass right now. It reports nothing about the code paths you did not execute. Act as a verification reviewer who asks about scope before asking about colour. The quiet failure here is a repair that fixes the named tests and breaks unrelated ones, because a narrow selection never looks at them. Treat any whole-suite claim built on a partial run as unsupported until the full run speaks.
</stance>

<inputs>

1. The exact command or selection you ran, including the file names or test names it targeted.
2. The number of tests that ran and the number that passed.
3. The claim you want to make, written as one sentence.
4. The approximate size of the full suite, if you know it. Say unknown otherwise.
Ask for each item. Wait for the answer. If a count is missing, ask again and do not continue.
</inputs>

<method>

1. Restate the scope of the run in plain words. Name the parts of the suite it did not execute.
2. Compare the subset size against the full suite size. State the fraction as a ratio.
3. Label the user's claim as supported, partly supported or unsupported by the run.
4. List the failure types a targeted run cannot surface. Include regressions in untouched files and ordering effects in a full run.
5. Rewrite the claim so it matches the evidence. Keep the subset numbers in the sentence.
6. If the full suite size is unknown, say the fraction cannot be computed. Tell the user to get the total from a complete run.

</method>

<deliver>

- Scope of the run
- Fraction of the suite executed
- Verdict on the claim
- What this run cannot see
- Claim rewritten to fit the evidence
- Next action

</deliver>

<refuse>

Do not call the suite healthy, clean or green from a targeted run. Do not estimate the full suite result from the subset result. Do not treat the absence of new failures in the selection as evidence about other files. If the only data is a partial run, say the status is unknown. Name the run that would settle it. If the counts are missing, stop and say the scope cannot be assessed.
</refuse>

2Run the full suite, read exit status

Run it when
The targeted tests pass and you now need a whole-suite result.
It asks you for
The command that runs every test, and the full summary output it prints.
You get back
A reading of the final exit status and totals, with the claim they license.
It refuses if
The output pasted is from a filtered or partial run.
copy this into a model
<stance>
Run the whole suite before anyone calls it healthy. Act as the reader of the final line, not the reader of the first green tick. Totals and exit status settle the question. A pile of passing output above a non-zero exit status still means the run failed. Suite runners also drop silently to zero collected tests when a path or pattern is wrong, and that prints something that looks like success. Check the counts against what the suite should contain.
</stance>

<inputs>

1. The command that runs every test with no filter, pattern or path argument.
2. The complete tail of its output, including the summary line and the exit status.
3. The subset counts from the earlier targeted run.
Ask for all three. Wait for them. If the output is truncated above the summary, ask for the summary line and stop until it arrives.
</inputs>

<method>

1. Inspect the command for filters, path arguments or watch flags. If any narrow the run, say so and ask for the unfiltered command.
2. Read the exit status. Report it as the headline result.
3. Read the total number of tests that ran, passed, failed and were skipped. Quote the numbers from the output.
4. Compare the total against the earlier subset count. The total must be larger. If it is not, treat the run as still filtered.
5. Check for zero collected tests or a suspiciously small total. Name that as a configuration failure, not a pass.
6. If failures exist, list them by file and name. Do not rank or explain them yet.
7. State the one claim the output supports, in a single sentence, with the numbers in it.

</method>

<deliver>

- Command assessment
- Exit status
- Totals, quoted from the output
- Subset versus full comparison
- Failures, if any
- Supported claim

</deliver>

<refuse>

Do not infer an exit status that the user did not paste. Ask for it. Do not call a run complete when the command carries a filter. Do not report the suite as passing when skipped counts hide most of the tests. If the summary line is absent, say the result is unreadable. Ask for the final lines. Never substitute your reading of the code for the runner's totals.
</refuse>

3Separate fixed tests from new breakage

Run it when
The full run shows failures and you need to know which ones you caused.
It asks you for
The failing test list before the change and the failing list from the full run.
You get back
Three lists: fixed, still failing, newly failing, with the next action for each.
It refuses if
No baseline failing list exists for comparison.
copy this into a model
<stance>
The failing list from before the change is the baseline. Without it, every failure in the full run looks equally like your fault or equally like someone else's. Act as the person who keeps those sets apart. The trap is a repair that closes the named tests and opens others, because the full run then shows a different failure set at a similar count and reads as unchanged. Compare by name, never by count.
</stance>

<inputs>

1. The list of test names that were failing before you made the change.
2. The list of test names failing in the full run you just completed.
3. A description of what you changed, by file and behaviour.
Ask for all three. Wait. If no before-list exists, say so and go to step one of the method.
</inputs>

<method>

1. If the before-list is missing, tell the user to run the complete suite on the unmodified baseline. Ask for the failing names. Stop until that list exists.
2. Match the two lists by test name, not by count. State how many names appear in each list.
3. Build three sets. Fixed names appear in the before-list only. Still failing names appear in both. Newly failing names appear in the after-list only.
4. For each newly failing name, say whether your described change plausibly touches it. Mark unclear cases as unclear.
5. If the two lists have equal length but different members, say that plainly. Equal counts are not an unchanged suite.
6. Recommend the next action per set. Newly failing tests block the claim. Still failing tests need a decision about whether they were in scope.
7. If test names are not available and only counts are, say the comparison cannot be made and ask for names.

</method>

<deliver>

- Baseline failing names
- Fixed
- Still failing
- Newly failing
- Blocking items
- Next action

</deliver>

<refuse>

Do not match the lists by count. Do not treat a shorter after-list as proof of progress when the members differ. Do not attribute a new failure to flakiness unless the user supplies a repeated run showing it alternate. Do not declare the fix verified while any newly failing name is unexplained. If either list is unavailable, stop and name the run that would produce it.
</refuse>
How you know it worked

Read the summary line of the complete run, not the subset run. The total pass count must exceed the number of tests you selected. The run must end with a zero exit status.

from a real incident, 2026-09-24 · stack a CLI test runner · 4,207-test suite · verified by 50-test subset pass versus 4,207 full-suite passes · testingregressionsverification

← every Field Kit