Field Kit

A model CLI can fail with exit zero

Validate the provider result as well as the process exit code before accepting a model answer.

What happened

A model CLI printed an API error on standard output and exited successfully; its caller treated that error sentence as an accepted answer until the first non-empty line was checked.

2026-09-15 TypeScript · CLI provider · text output measured: Failure prefix, benign mentions and caller behavior

2 prompts, in order

Read the answer before accepting it A provider returns an error sentence as though it completed the task.
Preserve legitimate error discussion A new failure detector rejects narrated errors in CLI output.

1Read the answer before accepting it

Run it when: A provider returns an error sentence as though it completed the task.
It asks you for: The exact standard output, standard error, exit code, installed CLI behavior and provider caller.
You get back: The output condition that separates a failed call from a legitimate answer.
It refuses if: Do not infer successful model work from the process exit code alone.

copy this into a model

<stance>
Your job is to separate an observed failure from a convincing explanation that has never been checked. Keep the reader's actual system in view. A result is useful when another person can repeat the observation and get a different answer if the proposed repair is wrong.
</stance>

<inputs>
Ask for these before drawing a conclusion. Wait for anything essential that is missing.
1. The exact standard output, standard error, exit code, installed CLI behavior and provider caller.
2. The exact user-visible symptom, including the original output and the time it was observed.
3. The implementation and launch conditions responsible for that output. Read the relevant source in full.
4. What may be changed safely, and which actions require the owner's approval.
5. Any existing verification that could distinguish the proposed cause from the alternatives.
</inputs>

<method>
1. Capture the original output and exit code from the reported failure. Determine whether the CLI itself returned success while narrating the failed request in its text output.
2. Read where the caller converts output into an answer. Identify the exact point at which the failure sentence becomes accepted work or enters an agent's report.
3. Derive the recognition rule from the installed provider's actual output. Limit it to the discriminating position and shape, such as the first non-empty line, so ordinary prose about errors can still pass.
4. Define the failed-turn outcome and preserve a safe diagnostic. Keep authentication, rate limits and upstream failure separate where the output supports them; never invent the reason from an empty response.
</method>

<deliver>
- OBSERVED: the behavior actually established, with the output that supports it.
- EXPLANATION: what the observation settles and what remains unknown.
- NEXT ACTION: one bounded action, its owner, and its expected result.
- PROOF: the observation that would establish success, plus the result that would disprove it.
</deliver>

<refuse>
Do not infer successful model work from the process exit code alone.
Do not invent measurements, defaults, source contents, or a successful run. Keep historical results separate from today's observations. Never transmit private inputs or perform a paid or external action just to complete this check. If access or authorization is missing, stop and name the exact missing condition.
</refuse>

2Preserve legitimate error discussion

Run it when: A new failure detector rejects narrated errors in CLI output.
It asks you for: The pinned failure rule, existing provider tests and benign answers mentioning the same words.
You get back: Failure rejection, benign-answer acceptance and proof that the caller handles both results correctly.
It refuses if: Do not broaden the detector until every answer containing the word error is rejected.

copy this into a model

<inputs>
Ask for these before drawing a conclusion. Wait for anything essential that is missing.
1. The pinned failure rule, existing provider tests and benign answers mentioning the same words.
2. The exact user-visible symptom, including the original output and the time it was observed.
3. The implementation and launch conditions responsible for that output. Read the relevant source in full.
4. What may be changed safely, and which actions require the owner's approval.
5. Any existing verification that could distinguish the proposed cause from the alternatives.
</inputs>

<method>
1. Use the existing provider tests to feed a narrated failure through the actual caller with a successful process exit. Verify the turn fails before the text can be stored as the model's answer.
2. Check leading whitespace and empty lines using the existing cases. Recognition should track the first actual output line rather than an incidental blank line.
3. Pass a legitimate answer that quotes or explains the same failure phrase elsewhere. It must remain an answer; a detector that rejects useful debugging prose creates a new defect.
4. Recheck the rule against the installed CLI's documented or observed output when available. If that observation cannot be made, mark the pin unverified while reporting the isolated behavior checks that actually ran.
</method>

<refuse>
Do not broaden the detector until every answer containing the word error is rejected.
Do not invent measurements, defaults, source contents, or a successful run. Keep historical results separate from today's observations. Never transmit private inputs or perform a paid or external action just to complete this check. If access or authorization is missing, stop and name the exact missing condition.
</refuse>

How you know it worked

A provider failure on the first non-empty line must fail the turn even when the process exits successfully. A legitimate answer that discusses the same error text must remain valid. Pin the recognition rule to the actual provider output shape.

← every Field Kit