Passante A Passante publication

En Passante

In Passing
Field Kit

The checker flagged the frontmatter, not the prose

A checker earns a place in live configuration only after it runs against real structured files, a synthetic payload of the real shape, and every outcome it claims to produce.

What happened

On 2026-08-12, a text checker headed for a live hook reported violations inside YAML frontmatter and fenced code blocks, and the safety rules behind it cleared only after all 14 scenarios ran.

2026-08-12 a text checker on a JSON-configured hook measured: 14 safety scenarios: block, warn, allow, override, fail-open
3 prompts, in order
  1. Find false positives on real files A text checker is written and not yet wired to a hook.
  2. Build a synthetic payload before wiring Hook logic must read data from an external system or API.
  3. Exercise every outcome a rule claims Safety rules are documented and nobody has watched them fire.

1Find false positives on real files

Run it when
A text checker is written and not yet wired to a hook.
It asks you for
The run command, at least six real files, and the rules as stated.
You get back
A findings table, false positives grouped by cause, and fixture text to paste.
It refuses if
The checker has not been run on real files.
copy this into a model
<stance>
The checker is the thing under test here, not the document. A gate that reports findings inside YAML frontmatter or a fenced code block is wrong about the text it reads, and its output stops being read. Structured markup looks like ordinary prose to a naive sentence splitter. That is the trap. Your task is to make the checker fail now, in a terminal, on content it will actually meet, where each failure costs nothing.
</stance>

<inputs>

1. The command that runs the checker on one file.
2. Paths to at least six real files the checker will see in production.
3. The rules the checker enforces, written as the author states them.
4. Whether the checker is already attached to a live hook.
Ask for all four. Wait for each answer. Do not guess a command. Do not invent file contents.
</inputs>

<method>

1. Run the checker on each real file. Record file, line and finding.
2. Classify every finding. Mark it a true violation or a false positive.
3. Name the structure behind each false positive. Candidates include frontmatter, fenced code, tables, lists, inline code and URLs.
4. Group false positives by cause. One exclusion rule often clears several findings.
5. Propose the narrowest fix per cause. Prefer skipping a region over loosening a rule.
6. Add every failing input to a fixture file. Paste the exact text that broke the checker.
7. Re-run the checker on the fixture and the real files. Confirm false positives disappear and true violations remain.
8. If a finding is ambiguous, say so. Ask the author to rule on it.
</method>

<deliver>

- Findings table: file, line, finding, verdict.
- False positives grouped by cause.
- Proposed exclusions, narrowest first.
- Fixture additions, as text to paste.
- Ambiguous cases, listed as open questions.
</deliver>

<refuse>

- Refuse to approve wiring while any listed real file remains unrun. Name the missing files.
- Refuse to report a clean pass from reading the checker source. Only executed runs count.
- Refuse to delete or weaken a rule to clear a false positive. Ask the author first.
- Stop if the run command errors. Report the error text and the exit code.
- Stop if every real file returns zero findings. Say the gate has not been exercised.
</refuse>

2Build a synthetic payload before wiring

Run it when
Hook logic must read data from an external system or API.
It asks you for
One real captured payload or the observed fields, plus the intent in one sentence.
You get back
A synthetic payload file, variant results, and the configuration entry as a diff.
It refuses if
A field the logic depends on has never been observed.
copy this into a model
<stance>
A hook reads whatever the outside system hands it, not what a schema document promises. Field names drift. Values arrive empty. The logic that looks correct in isolation reaches the wrong decision the first time it meets real shape. Hold the live configuration file closed until a payload built from observed fields drives the logic to the decision the author named.
</stance>

<inputs>

1. One real captured payload, or the exact field names the author has observed.
2. The decision the hook is meant to reach, in one sentence.
3. The configuration file the hook entry would join.
4. The command that runs the logic against a file or a stream.
Ask for all four. Wait for each answer. Treat an unanswered item as a stop, not a default.
</inputs>

<method>

1. List every field the logic depends on. Mark each as observed or assumed.
2. If a dependent field is only assumed, stop. Ask for one real sample.
3. Build a synthetic payload matching the observed shape. Keep field names exactly as observed.
4. Build variants. Cover the happy path, a missing field, an empty value, an extra field and a wrong type.
5. Run the logic on each variant outside the live configuration. Record the decision reached.
6. Compare each decision with the one sentence of intent. Note every mismatch.
7. Add one variant that must produce the opposite decision. Confirm the logic can say no.
8. Once every variant matches intent, write the configuration entry as a diff. Do not apply it.
</method>

<deliver>

- Field table: name, observed or assumed, used by the logic.
- Synthetic payload, as a file to save.
- Variant results: input, expected decision, actual decision.
- Mismatches, each with the smallest change that fixes it.
- Proposed configuration entry, as a diff.
</deliver>

<refuse>

- Refuse to edit the live configuration file. Output a diff instead.
- Refuse to build a payload from field names nobody has observed. Ask for a sample.
- Refuse to call the logic verified when only the happy path ran. Name the untested variants.
- Stop if every variant passes, including the one designed to fail. Report that the harness misses the logic.
- Stop if the author cannot state the decision in one sentence. Ask for that sentence first.
</refuse>

3Exercise every outcome a rule claims

Run it when
Safety rules are documented and nobody has watched them fire.
It asks you for
The rule list with claimed outcomes, the live configuration, and the trigger command.
You get back
A coverage matrix, documented-only rules, scenario results, and untriggered scenarios.
It refuses if
A rule has no matching entry in the live configuration.
copy this into a model
<stance>
Documentation blocks nothing. The mechanism does the blocking, and the mechanism lives in configuration that may not contain the entry the rule assumes. A rule can read as strict and still allow everything, because the path that enforces it was never wired. Treat each claimed outcome as a hypothesis. Block, warn, allow, override and fail-open are separate behaviours, and each needs its own scenario before anyone calls the set active.
</stance>

<inputs>

1. The rule list, with the outcome each rule claims.
2. The live configuration file meant to carry the mechanism.
3. The command that triggers a rule evaluation.
4. Whether any rule has ever been seen to fire.
Ask for all four. Wait for each answer. Do not infer a configuration path.
</inputs>

<method>

1. Read the live configuration. Confirm the entry that implements each rule exists.
2. Mark any rule with no matching entry as documented only. Report it before testing.
3. Write one scenario per claimed outcome per rule. Name the input and the expected outcome.
4. Count the scenarios. Report the total before running any of them.
5. Run each scenario. Record the actual outcome beside the expected one.
6. Flag every mismatch. A rule that warns when it claims to block is a defect.
7. Test the override path on its own. Confirm it demands an explicit action.
8. Test fail-open deliberately. Remove the dependency the mechanism needs and record what happens.
9. If a scenario cannot be triggered, say so. Name what is missing to trigger it.
</method>

<deliver>

- Coverage matrix: rule, claimed outcome, scenario, result.
- Documented-only rules, with the missing configuration entry named.
- Mismatches between claimed and actual outcomes.
- Override and fail-open results, reported separately.
- Untriggered scenarios, each with the blocker.
</deliver>

<refuse>

- Refuse to count a rule as active because a document describes it. Require an executed scenario.
- Refuse to infer an outcome from reading configuration. Run the trigger command.
- Refuse to summarise the set as working while any scenario stays untriggered. List those scenarios instead.
- Stop if the trigger command cannot reach the mechanism. Report the command and its output.
- Stop if a rule blocks when it claims to allow. Report it as a defect before continuing.
</refuse>
How you know it worked

Run the checker on a file holding only YAML frontmatter and one fenced code block. The report lists zero findings. Then run every scenario and confirm each prints the outcome its rule claims.

from a real incident, 2026-08-12 · stack a text checker on a JSON-configured hook · verified by 14 safety scenarios: block, warn, allow, override, fail-open · hooksvalidationfalse-positivestesting

← every Field Kit