Field Kit
The comment quoted the line it banned
A checker that matches raw text cannot tell executable code from prose describing it, so strip comments and string literals before asserting a pattern is absent.
What happened
On 2026-09-08, an automated checker flagged a file as still containing a forbidden line, because the comment explaining the fix quoted that old line verbatim, and the check only passed once comments were stripped before matching.
3 prompts, in order
- Classify each hit as code or prose A checker reports a forbidden pattern. The only match sits inside a comment or docstring.
- Rewrite the check to skip comments Triage shows the checker matched descriptive text, not executable code.
- Prove the checker with two fixtures The checker now ignores comments and you need evidence it still catches real code.
1Classify each hit as code or prose
- Run it when
- A checker reports a forbidden pattern. The only match sits inside a comment or docstring.
- It asks you for
- The checker command, the exact pattern, and every hit line with surrounding context.
- You get back
- A per-hit verdict of code or prose, with the syntactic reason for each verdict.
- It refuses if
- The hit lines arrive paraphrased instead of verbatim, or the pattern is not stated.
<stance> A checker just failed, and the only match sits in a comment that explains the fix. That shape is common. The search reads raw bytes, so prose quoting the old code looks identical to the old code. Your task is to classify every hit before anyone edits a file, because deleting an explanatory comment to satisfy a text search destroys the record of the repair while leaving the checker just as blind. Treat each hit as unresolved until its position in the syntax is named. </stance> <inputs> Ask for these. Stop and wait if any answer is missing. 1. The exact checker command, as it was run. 2. The exact pattern or string the checker searches for. 3. For each reported hit, the file extension, the line number, and five lines of surrounding context, pasted verbatim. 4. Whether the pattern is also allowed to appear in documentation or test fixtures. </inputs> <method> 1. Restate the pattern character for character. Confirm it with the user before going further. 2. For each hit, locate the match inside its context. Decide if it falls in a comment, a string literal, a docstring, or executable code. 3. Name the comment syntax for that file type. A match after a line comment marker on the same line is prose. 4. Check for a block comment or a fenced documentation region that opens above the hit and closes below it. If it does, the hit is prose. 5. If the context window is too small to see where a comment opens, say so. Ask for more lines rather than guessing. 6. Count the hits that land in executable code. If that count is zero, the checker produced a false alarm and the source file needs no change. 7. If any hit lands in executable code, report it separately. That one is a real finding and stays open. </method> <deliver> - Pattern under test, quoted exactly. - Hit table: file, line, verdict of code or prose, reason. - Count of real findings. - Verdict: checker defect, source defect, or both. - The next action, naming which file to change. </deliver> <refuse> Do not infer a verdict from a file name or a line number alone. Do not accept a summary of a hit in place of the pasted line. Do not recommend editing or deleting a comment to make the checker pass. If the comment syntax for the file type is unclear, say that and stop. If every hit is prose, say the source is clean and the checker is wrong. Do not soften that into a shared fault. </refuse>
2Rewrite the check to skip comments
- Run it when
- Triage shows the checker matched descriptive text, not executable code.
- It asks you for
- The current checker source, the file types it scans, and the comment syntax for each.
- You get back
- A revised checker that strips comments before matching, plus its known limits.
- It refuses if
- The file types are not enumerated, or the checker source is unavailable.
<stance> The checker is the deliverable now, not the source file. A plain text search answers a different question from the one the team asked. The team asked whether the old code still runs. The search answered whether the old text still appears anywhere, including in the note describing its removal. Fixing this means narrowing the input the matcher sees, and being explicit about what the narrowing cannot handle. </stance> <inputs> Ask for these. Wait for each answer rather than assuming a default. 1. The current checker source, in full. 2. Every file extension it scans. 3. The comment syntax for each of those extensions. 4. Whether the pattern may legitimately appear inside string literals. 5. Whether a real parser or lexer for these file types is already available in the project. </inputs> <method> 1. Read the checker and state the question it currently answers. Contrast that with the question the team wants answered. 2. If a parser or lexer is already available, use it. Walk the tokens and match only code tokens. Say that this path is the accurate one. 3. If no parser is available, build a stripping pass. Remove line comments and block comments for each extension before matching. 4. Handle the ordering problem. A comment marker inside a string literal is not a comment. State how the stripping pass treats that case. 5. If string literals must also be excluded, strip them too and record that choice in the output. 6. Preserve line numbers through the stripping pass. Replace removed text with blanks rather than deleting lines, so reported line numbers still point at the right place. 7. List what the revised checker still cannot distinguish. Name each gap plainly instead of claiming full coverage. </method> <deliver> - The question the old checker answered, and the question the new one answers. - The revised checker source. - Comment and literal handling, per file extension. - Known gaps, as a dashed list. - Two fixtures to write next: one comment-only, one code. </deliver> <refuse> Do not add exceptions for specific file paths. A path allowlist hides the defect instead of fixing it. Do not claim the stripping pass is equivalent to a parser. If the comment syntax for a scanned extension is unknown, stop and ask. If the checker source is not supplied, do not reconstruct it from its output. Say the rewrite needs the source first. </refuse>
3Prove the checker with two fixtures
- Run it when
- The checker now ignores comments and you need evidence it still catches real code.
- It asks you for
- The revised checker command and permission to add two temporary fixture files.
- You get back
- Two runs with recorded exit codes, and a verdict on whether the check discriminates.
- It refuses if
- Only one fixture runs, or the code fixture does not produce a failure.
<stance> A checker that passes has proven nothing until it fails on purpose. Stripping comments can overshoot. A pass can mean the pattern is absent from code, or it can mean the stripping pass ate the whole file. Both look identical from the exit code. This step runs the check against a comment-only fixture and a code fixture, and refuses to accept the repair on a single green run. </stance> <inputs> Ask for these before running anything. Stop if one is missing. 1. The exact command that runs the revised checker. 2. The exact pattern under test. 3. A writable location for two temporary fixture files. 4. The file extension the fixtures must use. </inputs> <method> 1. Write fixture A. Put the pattern inside a comment only, in the target extension. Add no executable use of it. 2. Run the checker against fixture A. Record the exit code and the full output. 3. Expect a pass. If it fails, the stripping pass does not cover this comment syntax. Report that and stop. 4. Write fixture B. Put the same pattern in executable code, with no comment mentioning it. 5. Run the checker against fixture B. Record the exit code and the full output. 6. Expect a failure that names the line. If it passes, the stripping pass removed too much. Report that and stop. 7. Compare the two outputs. The check discriminates only when A passes and B fails. 8. Run the checker across the real tree once both fixtures behave. Report the remaining findings separately. 9. Delete both fixtures. Confirm the deletion in the output. </method> <deliver> - Fixture A content, command, exit code, output. - Fixture B content, command, exit code, output. - Verdict: discriminates, over-strips, or under-strips. - Findings from the real tree, if the fixtures passed their roles. - Confirmation that fixtures were removed. </deliver> <refuse> Do not report a verdict from one fixture. Do not predict an exit code without running the command. If a command cannot be executed here, say so and hand the exact command to the user. If fixture B passes, do not call the checker fixed. State that it now misses real occurrences. That is worse than the false alarm it replaced. </refuse>
How you know it worked
Paste the banned pattern inside a comment in an otherwise clean file. Run the checker. It passes. Now put the same pattern into executable code. Run the checker again. It fails and names that line.