Field Kit
The audit reported one of five missing
An audit that stops at the first failure reports the first failure, not the extent of one. A count you did not verify by checking every item is a lower bound wearing the appearance of a total.
What happened
An audit reported a single missing marker. Checking all seven by hand showed 5 of 7 were gone — and the same run surfaced a check that passed a double-probe twice, then failed the first real sweep.
2 prompts, in order
- Check whether your audit counts or stops An audit reports a small number of problems and you are about to fix exactly those.
- Run the real sweep before trusting the probe A check has passed its isolated probe and you are about to wire it into automation.
1Check whether your audit counts or stops
- Run it when
- An audit reports a small number of problems and you are about to fix exactly those.
- It asks you for
- The audit's output, the full set of items it should check, and whether it exits on first failure.
- You get back
- The true count, and whether past passes from this audit meant anything.
- It refuses if
- You supply the reported count without the full item set. The gap between them is what is being measured.
<stance> You have fixed the one problem an audit reported, watched it go green, and found four more of the same problem still sitting there. The audit was not wrong. It stopped at the first failure and said so in a way that read like a total, and everyone downstream did the arithmetic the reading invited. So you check the full set by hand once, before trusting any number a checker gives you. </stance> <inputs> 1. The audit's output, verbatim. 2. The complete set of items it is supposed to verify. 3. Whether it returns on the first failure or accumulates. 4. What its passing output looks like — does it report how many items it inspected. 5. How long it has been in use. </inputs> <method> 1. Read the output literally. A message naming one item is evidence about one item; whether it is evidence about the rest depends entirely on whether the loop continued, and the message usually will not say. 2. Check every item in the set by hand, once. This is tedious and it is the only thing that establishes the true count. Report both numbers side by side: what the audit said, what is actually true. 3. If they differ, say what that means about the audit's history. Every past green from a first-failure checker means the first item passed, and nothing more. That is a smaller claim than anyone has been reading. 4. Recommend accumulation over early return, and require the passing output to state how many items it inspected. A green with no count cannot be distinguished from a green that examined nothing, and both look identical in a log. 5. Note where two rules contradict each other. Where one instruction forbids exactly what another requires, a change satisfying the first silently breaks the second, and no individual reviewer is wrong. That is a reconciliation problem, not a compliance problem, and treating it as the latter produces a fix that breaks again. 6. Say what to do when reconciling: find the form that satisfies both, and prefer it over retargeting a check at something outside the repository, which usually makes the check unrunnable in automation. </method> <deliver> - REPORTED VERSUS ACTUAL — both counts, side by side - STOPS OR ACCUMULATES — which, and how you can tell - WHAT PAST GREENS MEANT — restated honestly - INSPECTED COUNT — whether the passing output states one - CONTRADICTORY RULES — any pair, and the form satisfying both </deliver> <refuse> - Do not accept a reported count as a total without checking the full set once. - Do not treat a green from a first-failure checker as coverage. Say what it actually establishes. - Do not approve a passing output with no inspected count. - Where two rules contradict, do not pick one and call it fixed. Say both must hold and find the form that does. </refuse>
2Run the real sweep before trusting the probe
- Run it when
- A check has passed its isolated probe and you are about to wire it into automation.
- It asks you for
- How the probe invoked it, how the sweep invokes it, and whether the real sweep has been run.
- You get back
- Whether the probe and the sweep exercise the same thing, and what the real run found.
- It refuses if
- The real sweep has not been run. It will say the probe is not the evidence and name the run that is.
<stance> You have wired a check that passed its probe twice in a row and watched it fail the first real sweep. The probe was not broken. It ran the check in a cleaner environment, in a different order, with less around it — and every one of those differences is where a check finds out what it actually depends on. A probe tells you a check can pass. Only the sweep tells you it will. </stance> <inputs> 1. How the probe invokes the check: directory, environment, arguments, isolation. 2. How the sweep invokes it: the same list. 3. Whether the real sweep has been run with this check in it, and what happened. 4. What else runs before it in the sweep, and whether anything shares state. 5. Its runtime under each. </inputs> <method> 1. Ask whether the real sweep has been run. If not, stop and say the probe is not the evidence. This costs one run and it is the difference between wiring a check and wiring a flake. 2. Compare the two invocations field by field. Working directory, environment variables, argument list, and what has already run. Every difference is a hypothesis about why the check might behave differently. 3. Pay particular attention to shared state. A check that passes alone and fails in sequence is usually reading something an earlier step wrote, and it is reporting that honestly. 4. Say what a probe legitimately establishes: that the check runs, that it is deterministic across repeats, and roughly what it costs. Those are real and they are not the same as fitness for the sweep. 5. Recommend probing more than once AND running the real sweep before wiring, as two separate requirements rather than alternatives. One establishes determinism; the other establishes compatibility. 6. If the check failed the real sweep, treat that as the check working. It found a genuine dependency, and the fix belongs upstream rather than in the check's own logic. </method> <deliver> - SWEEP RUN? — whether it has, and if not, stop here - INVOCATION DIFFERENCES — field by field - SHARED STATE — anything an earlier step writes that this reads - WHAT THE PROBE ESTABLISHED — stated honestly, and what it did not - IF IT FAILED — read as the check working, with the fix placed upstream </deliver> <refuse> - Do not wire a check on probe results alone. Say the real sweep is the evidence. - Do not treat a sweep failure as a defect in the check before looking for shared state. - Do not accept a single probe as proof of determinism. Two, minimum. - Do not fix a real dependency by loosening the check. Say where the fix belongs. </refuse>
How you know it worked
Check every required item by hand once and compare the total against what the audit reported. If the audit said one and the manual pass finds five, the audit is a first-failure detector and every past green from it means less than it appeared to.