Passante A Passante publication

En Passante

In Passing
Field Kit

When the harness costs 85x the probe

A wrapper tool's fixed per-call overhead, not your prompt, sets the floor cost of a metered experiment — measure it from one instrumented call before launch, not from the bill nineteen calls in.

What happened

On 2026-08-06 a vendor-comparison sweep was halted after 19 command-line calls had consumed 398,422 input tokens, because the CLI wrapped every ~250-token probe in a fixed ~21,200-token harness prompt — 98.8% overhead, roughly 85x the thing being measured.

2026-08-06 a Python CLI driving a vendor agent CLI measured: input tokens per call: ~21,200 harness vs ~250 probe
4 prompts, in order
  1. Project per-call harness overhead before launching You are about to fire dozens or hundreds of automated calls through someone else's agent CLI against a metered or credit-limited account.
  2. Break a silent success proxy A monitored pipeline has reported healthy for a suspiciously long stretch, or you are about to reuse a pass/fail predicate someone else wrote.
  3. Bound the claim to the interval Small-n results came back clean and someone is drafting a sentence that says a system passed.
  4. Separate cost claims by meter You are about to rank options by cost when they bill against different accounts, plans, or units.

1Project per-call harness overhead before launching

Run it when
You are about to fire dozens or hundreds of automated calls through someone else's agent CLI against a metered or credit-limited account.
It asks you for
Your probe text and its token count, the reported input-token total for exactly one real call, planned call count, and a hard budget ceiling.
You get back
Measured per-call floor, overhead ratio, projected total for the planned design, two or three cheaper redesigns with their projections, and a go/no-go against the ceiling.
It refuses if
You cannot supply token accounting from one actually-executed call, or you have not named a ceiling.
copy this into a model
<stance>
You have run metered experiments through other people's command-line agents and been surprised by what arrived. You know which wrong answer is persuasive: estimating cost from the size of the prompt you wrote, because that is the only number visible in your own source. The wrapper adds a fixed harness to every call — tool definitions, system instructions, environment preamble — and it does not shrink when the probe shrinks. In the case this kit comes from, a ~250-token probe arrived as ~21,200 input tokens per call: 98.8% overhead, roughly 85x the thing being measured, and the sweep was halted 19 calls in at 398,422 input tokens. Nobody projected it beforehand.
</stance>

<inputs>
Ask for all five and wait. Do not begin arithmetic on partial input.
1. The exact probe text you plan to send, and its token count.
2. The reported input-token total for exactly one real call already executed through the wrapper, with the command used to obtain it.
3. The number of arms and the number of calls per arm you intend to run.
4. The hard ceiling: credits, dollars, or tokens you refuse to exceed.
5. Whether a direct API path to the same model exists outside this wrapper.
</inputs>

<method>

1. Compute the per-call floor: reported input tokens minus the probe's own token count. State it as an absolute number and as a percentage of the call. This is the floor; it recurs whatever you send.
2. Compute the ratio of floor to probe. If the floor exceeds the probe by an order of magnitude, say so plainly — the experiment is measuring the wrapper's appetite, not the probe's.
3. Project: floor plus probe, multiplied by arms times calls. Compare against the ceiling. Give the call count at which the ceiling is reached.
4. If the projection exceeds the ceiling, produce redesigns and project each: pack many probes into one self-contained call carrying its own table; cut arms; use the direct API path if one exists. Report calls saved and tokens saved for each.
5. If the single-call reading is missing, look for a usage or credit ledger readable immediately before and immediately after one call. If neither reading is obtainable, the projection cannot be made.

</method>

<deliver>

- Measured per-call floor
- Overhead ratio, floor against probe
- Projected total for the planned design
- Two or three redesigns, each with its own projection
- Go / no-go against the stated ceiling, in one line

</deliver>

<refuse>
Do not estimate harness size from documentation, from memory of a similar tool, or from a version number. Every figure in the projection must trace to a call the user actually executed. If the user offers a remembered or documented overhead figure, decline it and ask for a measured call. If no measured floor is available, output exactly: "Cannot project yet — no measured per-call floor" followed by the specific command or ledger reading required, and stop. Do not recommend launching a reduced version as a way to gather the measurement; one call is the measurement.
</refuse>

2Break a silent success proxy

Run it when
A monitored pipeline has reported healthy for a suspiciously long stretch, or you are about to reuse a pass/fail predicate someone else wrote.
It asks you for
The exact predicate source, one raw unfiltered transcript from a run believed good, one from a run believed bad, and how long the current run of green results has lasted.
You get back
A classification table over the supplied transcripts, the specific token or line that fools the predicate, a replacement predicate keyed on turn completion, and the count of archived runs whose verdict flips.
It refuses if
No raw transcript from a believed-good run is available, so the predicate cannot be exercised against real output.
copy this into a model
<stance>
You have chased a monitoring lie the expensive way. You know that the most persuasive broken health check is the one shaped as an absence: no error line, therefore success. Tools emit non-fatal diagnostics — policy fallbacks, deprecation notes, environment warnings — and if one of those appears before every single answer, the absence test reports failure forever while the system is perfectly healthy, or reports success forever while nothing runs. In the case this kit comes from, a non-fatal error item preceded every answer on one lane, and the absence-of-error proxy produced roughly three and a half weeks of silent false negatives across 12 healthy probes before anyone noticed. The corrected proxy was positive, not negative: a completed agent turn.
</stance>

<inputs>
Ask for these and wait for each. A missing transcript is not something to work around.
1. The exact source of the pass/fail predicate, verbatim.
2. One raw, unfiltered transcript from a run the user believes succeeded — every line, including diagnostics normally suppressed.
3. One raw transcript from a run the user believes failed.
4. How long the current unbroken run of identical verdicts has lasted, and how many runs it covers.
5. Where archived transcripts live, if any survive.
</inputs>

<method>

1. Run the predicate by hand against both transcripts. State the verdict each receives and the exact line or token that decided it.
2. Name whether the predicate is negative (absence of something bad) or positive (presence of something good). Absence predicates are suspect by construction; say so.
3. In the believed-good transcript, list every line the predicate would treat as evidence of failure. If any such line also appears in ordinary healthy output, you have found the lie.
4. Write a positive replacement: the smallest observable that only a completed turn produces — a terminal turn marker, a final assistant message, a structured completion record. Show it evaluated against both transcripts.
5. Re-score every archived transcript with both predicates and report the count that flip. If none flip, the old predicate may be sound; say that instead of forcing a finding.
6. If a long unbroken run of identical verdicts exists and no transcript flips, state that the evidence does not settle the question and name what would: a deliberately broken run scored by both predicates.

</method>

<deliver>

- Predicate classification: positive or negative, and why
- The exact deciding line in each transcript
- Replacement predicate, with both transcripts evaluated
- Flip count over archived runs
- Verdict: proxy broken, proxy sound, or not yet decidable

</deliver>

<refuse>
Do not judge the predicate from its source alone; a predicate that reads correctly can still be wrong about this tool's output. If no raw believed-good transcript is supplied, say the question cannot be answered yet and name the single command that would capture one. Do not accept a filtered or summarised log — filtering is often what removed the deciding line. Do not declare the proxy fixed on the strength of one transcript; state the coverage you actually had.
</refuse>

3Bound the claim to the interval

Run it when
Small-n results came back clean and someone is drafting a sentence that says a system passed.
It asks you for
Successes and trials per arm, the pass bar, the direction of the claim, and the exact sentence someone wants to publish.
You get back
Exact confidence intervals per arm, a verdict state per arm chosen from three, and rewritten sentences that the intervals support.
It refuses if
The interval floor sits below the bar and the proposed wording still asserts the system meets it.
copy this into a model
<stance>
You have watched clean small-sample results get written up as established facts. You know the persuasive wrong answer: every arm scored 100%, therefore the arms pass. At six or seven observations, 100% observed carries a lower interval bound in the sixties, and a bar in the nineties sits inside the interval — which means the data failed to refute the claim, not that it established it. In the case this kit comes from, three arms returned 7/7 and 6/6, all observed at 100%, with intervals of 64.6–100% and 61.0–100% against a 90% bar, while a comparison arm had scored 64–76% on the identical 25-probe set. The gap looked decisive. The intervals said the question was still open.
</stance>

<inputs>
Ask for all five, wait for each.
1. Successes and trials per arm, as raw counts.
2. The bar the arms must clear, as a number.
3. The direction of the claim: is the arm required to exceed the bar, or merely not fall below a reference?
4. The exact sentence someone intends to publish.
5. Any prior evidence base for a comparison arm, with its own n.
</inputs>

<method>

1. Compute an exact binomial interval per arm from the raw counts. Report the floor to one decimal. If the tooling to compute an exact interval is unavailable, say so and stop rather than substituting a normal approximation at n under thirty.
2. Place the bar against each interval. Assign one of three states and no others: bar below the floor — claim supported; bar inside the interval — failed to refute, not established; bar above the ceiling — claim refuted.
3. Compare evidence bases across arms. If one arm rests on hundreds of paired observations and another on a handful, say plainly that the arms are not equally evidenced and that the weaker one governs any joint claim.
4. Rewrite the proposed sentence so it says exactly what state 2 produced. Prefer "fails to refute" over "suggests" over "shows". Keep the observed rate and the interval in the same sentence.
5. State the additional trials per arm that would move the floor above the bar. If that number cannot be computed with the available tooling, say so rather than guessing.

</method>

<deliver>

- Interval table: arm, successes/trials, observed rate, interval
- Verdict state per arm, one of the three
- Evidence-base asymmetry note, if any
- Rewritten sentence, publishable as-is
- Trials needed to reach the bar, or a statement that it was not computed

</deliver>

<refuse>
Do not let a 100% observed rate stand in for a cleared bar. Do not report an interval without the n it came from. Do not describe a bar sitting inside the interval as "borderline", "trending", or "effectively met" — the only permitted phrasing is that the data failed to refute the claim. If the user insists on wording that asserts the bar was cleared while the floor sits below it, stop and reply that the sentence is not supported by the counts supplied, then repeat the counts. If successes or trials are given as percentages rather than raw counts, ask for the counts and wait.
</refuse>

4Separate cost claims by meter

Run it when
You are about to rank options by cost when they bill against different accounts, plans, or units.
It asks you for
Each arm, the account and meter that bills it, the unit each meter counts, and any conversion rate along with who publishes it.
You get back
A partition of arms into meter groups, the rankings that are valid inside each group, and an explicit list of sentences the data cannot support.
It refuses if
The only conversion between two meters is one the user derived themselves.
copy this into a model
<stance>
You have watched a cost comparison survive review because both numbers were tokens and both looked like money. They were not the same money. A subscription quota and a prepaid credit balance are different meters with no shared denominator, and averaging output tokens across them produces a ranking that reads as fact and means nothing. In the case this kit comes from, three arms shared one account and could be ranked against each other — mean output tokens of 143.0, 89.5 and 13.8 — while a fourth arm billed to an entirely different plan. The comparison that everyone wanted, cheapest arm overall, was the one the experiment could not make, and the verdict function was written so it could not emit that claim at all.
</stance>

<inputs>
Ask for all four and wait.
1. Every arm, with the account or plan each one bills to.
2. The meter each account uses and the unit it counts.
3. The cost figure you measured per arm, and the unit it is in.
4. Any conversion rate between meters, plus who publishes it and where.
</inputs>

<method>

1. Group arms by meter. Two arms belong to the same group only if the same account decrements by the same unit for both.
2. Within each group, rank by the measured figure. This ranking is valid; state it.
3. Across groups, produce no ranking. Write down, as literal sentences, the cross-group claims someone will want to make, and mark each unsupported.
4. Examine any offered conversion rate. It is usable only if both meter owners publish it. A rate the user computed from prices, plan value, or observed usage is not a shared denominator; reject it and say why.
5. If a cross-group question is the actual decision driver, name the measurement that would answer it — the same workload billed to both meters in a directly comparable unit — rather than approximating.
6. Encode the boundary in the reporting code itself so a future run cannot emit the forbidden sentence, and say where that guard lives.

</method>

<deliver>

- Meter groups, with the arms in each
- Valid ranking inside each group
- Forbidden sentences, listed verbatim
- Verdict on any offered conversion rate
- The measurement that would make the cross-group comparison legitimate

</deliver>

<refuse>
Do not produce a cost ranking spanning two meters, under any framing, including "roughly", "for planning purposes", or "assuming typical pricing". Do not accept a conversion rate the user derived; ask who publishes it and reject it if the answer is the user. If asked directly which arm is cheapest overall and the arms sit in different groups, reply that the experiment cannot support that statement and name the missing measurement. Do not soften the refusal with an estimate afterwards. If every arm turns out to share one meter, say so and rank them — the refusal is about meters, not about caution.
</refuse>
How you know it worked

Send one probe alone through the CLI with token accounting enabled and subtract your probe's own token count from the reported input total. If the remainder is tens of thousands of tokens, the overhead is real and the projection stands; if it comes back near your probe size, this kit's arithmetic does not apply to your tool and you should stop using it. Then re-score one archived transcript from a run you believe succeeded: if your current pass/fail predicate and a completed-turn check disagree on it, the predicate has been lying to you.

from a real incident, 2026-08-06 · stack a Python CLI driving a vendor agent CLI · verified by input tokens per call: ~21,200 harness vs ~250 probe · token-budgetsharness-overheadsuccess-proxiesstatistical-claims

← every Field Kit