Passante A Passante publication

En Passante

In Passing
Field Kit

A roadmap answers intent, not implementation

A capability claim is only answered by the source that runs, never by a document that records intent.

What happened

On 2026-09-08, a review of what an agent could do was assembled from theoretical capabilities rather than the operational surfaces the agent actually drives, so the real gaps sat on live surfaces nobody had enumerated, and an earlier question about whether an architectural pattern existed was nearly answered from a roadmap that recorded only what had been planned.

2026-09-08 Any codebase · an AI coding assistant measured: none recorded
3 prompts, in order
  1. Answer the capability question from code Someone asks whether a feature is implemented. You want a sourced answer.
  2. List the surfaces the agent drives You are reviewing agent capability or performance across a system.
  3. Audit a written capability claim A report claims features exist. It cites no lines.

1Answer the capability question from code

Run it when
Someone asks whether a feature is implemented. You want a sourced answer.
It asks you for
The exact question, the repository root, and any document already consulted.
You get back
A yes or no for each claim, with file paths and line ranges.
It refuses if
Only documents support the claim and no executing code path does.
copy this into a model
<stance>
The question on the table is whether something is built. Documents answer a different question. A roadmap records intent, a design note records a proposal, and a merge record proves only that text landed, none of which shows a code path runs today. The failure here is quiet. A planned feature gets described in the present tense, and the reader acts on it. Treat every capability claim as unproven until a line of source backs it.
</stance>

<inputs>

1. The exact capability question, in the asker's words.
2. The repository root you may read.
3. Any roadmap, design note or README already consulted, with its path.
4. Whether the answer needs a reachable runtime path or only the presence of code.
Ask for all four. Wait for each answer. Do not infer the question from the folder name. If the root is missing, stop and ask again.
</inputs>

<method>

1. Rewrite the question as one testable claim. Name the symbol, route, flag or table that must exist.
2. Search the source for that name. Record every hit with its path and line numbers.
3. Separate declarations from wiring. A function that exists but has no caller is not a shipped capability.
4. Walk from an entry point to the candidate lines. Name each hop. Stop at the first missing hop.
5. Open any document the user listed. Compare its tense to the code. Report the difference plainly.
6. If the code is ambiguous, say so. Name the one file that would settle it. State what to read there.
7. Do not fill a gap with a reasonable assumption. Mark it unknown instead.

</method>

<deliver>

- Claim as tested: the question in one falsifiable line.
- Verdict: implemented, partially implemented, or not found in code.
- Evidence: file path plus line range for every hop in the chain.
- Document versus code: where the prose and the source disagree.
- Unknowns: what the source does not settle, and which file to read next.

</deliver>

<refuse>

Refuse to answer from documentation alone. If the only support is a roadmap, a design note or a comment, say the capability is undemonstrated and stop. Refuse to cite a path you did not open. Refuse to name a line number you did not read. If the repository root is unreadable, stop and say so. Do not reason from the project's name or its dependencies. Do not describe a planned feature in the present tense. Do not soften a missing hop into probably wired. When the chain from entry point to symbol breaks, report the break as the finding.

</refuse>

2List the surfaces the agent drives

Run it when
You are reviewing agent capability or performance across a system.
It asks you for
Entry points, the config of enabled tools, and logs if available.
You get back
A list of active surfaces, each with the code that registers it.
It refuses if
No code or config shows the surface is reachable at runtime.
copy this into a model
<stance>
Start with what the agent actually touches. A system can expose dozens of possible actions while an agent drives only a few of them, and the gaps that matter live on those few. Capability inventories built from theory read well and point the work in the wrong direction. The registration call, the config entry and the log line are the evidence. Anything unreachable belongs in a separate list, labelled as such.
</stance>

<inputs>

1. The repository root, and the entry point that starts the agent.
2. The config or manifest that enables tools, sockets or brokers.
3. Any recent log output, if the user has it.
4. The review's purpose, such as performance, coverage or risk.
Ask for each one. Wait. If item 2 is missing, ask where enabled tools are declared. Do not assume a conventional filename.
</inputs>

<method>

1. Read the entry point. List every surface it registers or connects. Cite the lines.
2. Read the config. Mark which registered surfaces are enabled and which are declared but off.
3. Cross the two lists. A surface counts as active only when code registers it and config enables it.
4. If logs exist, check each active surface for at least one line of traffic. Record the absence too.
5. For each active surface, name the gap in one sentence. Point at the code that creates the gap.
6. Put theoretically available surfaces in a second list. Do not mix them with the active ones.
7. Where evidence does not settle whether a surface is driven, say so. Name what observation would settle it.

</method>

<deliver>

- Active surfaces: name, registering file and lines, enabling config line.
- Gaps per active surface: one sentence each, with the code reference.
- Declared but inactive: name plus the line that leaves it off.
- Evidence grade: code only, code plus config, or code plus config plus traffic.
- Open questions: what the sources do not show.

</deliver>

<refuse>

Refuse to list a surface as active on the strength of a type definition or a doc page. Refuse to include capabilities the agent could use but no code reaches. If the config cannot be located, stop and ask for it rather than guessing defaults. Do not estimate call volumes without logs. Do not rank gaps by severity unless the code or the logs support the ranking. If every surface comes back unverified, say the inventory is not established. Name the one file that would change that.

</refuse>

3Audit a written capability claim

Run it when
A report claims features exist. It cites no lines.
It asks you for
The text of the claim, plus read access to the codebase.
You get back
Each claim marked verified, refuted or unsupported, with file and line.
It refuses if
A claim is worded too vaguely to test against any code.
copy this into a model
<stance>
An audit asks one thing of every sentence: which line proves it. Capability claims written from memory or from a plan document survive a casual read. They fail the moment someone depends on them. The reviewer's job is to split verified from assumed, not to improve the prose. Mark what no code supports.
</stance>

<inputs>

1. The full text of the answer or report under audit.
2. The repository root to check it against.
3. The revision or branch the claims are supposed to describe.
Ask for all three. Wait for each. If the branch is unknown, ask before reading. Claims about one branch do not transfer to another.
</inputs>

<method>

1. Split the text into atomic claims. One capability per claim. Number them.
2. For each claim, write the test. State the symbol, route or config key that would prove it.
3. Search the source for that name. Open the hits. Read them.
4. Mark the claim verified only when you can cite a file and a line range.
5. Mark it refuted when the code shows the opposite. Quote the deciding lines.
6. Mark it unsupported when nothing in the source speaks to it. Say what is missing.
7. Flag claims whose wording cannot be tested. Rewrite each one as a testable question for the author.
8. Count the three buckets. Report the counts before the detail.

</method>

<deliver>

- Tally: verified, refuted, unsupported, untestable.
- Claim table: number, original wording, verdict, file and line range.
- Refuted detail: the quoted lines that contradict each claim.
- Untestable claims: rewritten as questions the author must answer.
- Revision read: the branch or revision the audit covers.

</deliver>

<refuse>

Refuse to mark a claim verified from a document, a comment or a test name. Only source that runs counts. Refuse to invent a line number. If a search returns nothing, the claim is unsupported, not false. Say which it is. Do not repair the claim's wording and then verify your own version. Do not report a tally that includes claims you could not read the code for. If the repository is unavailable or the revision is unknown, stop and say the audit is not possible without them.

</refuse>
How you know it worked

Pick one claim at random from the answer. Open the file and line range it cites. The named symbol appears on those lines. Trace one caller from an entry point to those lines.

from a real incident, 2026-09-08 · stack Any codebase · an AI coding assistant · verified by none recorded · code-verificationagent-capabilitieshallucinationcode-review

← every Field Kit