Field Kit
Facts the prompt never stated get re-derived
Any truth a prompt leaves unstated will be rebuilt from the nearest field name, and the rebuild is where the wrong answer enters.
What happened
On 2026-09-08, an agent labelled accounts by deriving them from a provider field, and the labels came out wrong, because the prompt never stated the account asymmetry as a fixed fact.
3 prompts, in order
- Separate stated facts from derived ones The agent labels records wrong and the code looks correct.
- Write the do-not-re-derive facts block You know which truths the agent guessed and want them stated instead.
- Prove the facts block changed the answer The facts block is in place and you need evidence it bites.
1Separate stated facts from derived ones
- Run it when
- The agent labels records wrong and the code looks correct.
- It asks you for
- The full prompt, the wrong output, the visible field names, and the correct labels.
- You get back
- Two buckets of claims, plus the field behind each wrong label.
- It refuses if
- You cannot supply the actual wrong output or the correct labels.
<stance> The mislabeled row is the evidence. A reviewer of agent output works out which claims arrived in the prompt and which the model assembled from nearby data. Inference wears the costume of knowledge, especially when a field name sits close to the answer. A provider column next to an account column invites that shortcut, and the shortcut holds right up until the asymmetric case arrives. </stance> <inputs> Ask for these four, numbered, and wait for all of them. 1. The prompt or instruction file the agent received, pasted in full. 2. The wrong output, with the incorrect labels marked. 3. The data shape the agent could see. Field names and sample values. 4. The correct labels, stated by the person who knows them. If the output or the correct labels are missing, stop and ask again. Do not infer the data shape from field names alone. </inputs> <method> 1. Split every claim in the output into two buckets. Bucket A is stated somewhere in the prompt. Bucket B is not. 2. For each bucket B claim, name the field or pattern the model most likely read. Quote the field name exactly. 3. Compare each bucket B claim against the correct label. Mark every disagreement. 4. For each disagreement, write one line. Name the claim, the field it leaned on, and the truth. 5. Check whether the prompt already states a fact the output ignored. That is a different failure from a missing fact. 6. If a wrong claim sits in bucket A, a facts block will not fix it. Report that and stop. 7. Where you cannot tie a wrong label to any visible field, say the source is unclear. Do not fill the gap. </method> <deliver> - Stated versus derived: the two buckets, as lists. - Wrong labels, each with the field that produced it. - Truths the prompt assumed but never wrote down. - Verdict: missing facts, or stated facts ignored. </deliver> <refuse> Refuse to name a cause you cannot anchor to a supplied field name. Refuse to invent business rules, ownership models or data semantics the user did not state. Do not write the fix in this step. Do not rank the errors by severity without being asked. If the correct labels arrive with no person standing behind them, stop and ask who verified them. </refuse>
2Write the do-not-re-derive facts block
- Run it when
- You know which truths the agent guessed and want them stated instead.
- It asks you for
- Each verified fact, who verified it, and the derivation it replaces.
- You get back
- A facts block of flat declarative lines, each with its forbidden inference.
- It refuses if
- A proposed fact has no owner or no derivation it rules out.
<stance> The facts block exists to remove guesses, one at a time. The author of such a block names each verified truth, the person who verified it, and the inference it forbids. Generic advice is not a fact. A line reading "be careful with accounts" steers nothing, while a line reading "accounts are asymmetric, never derive the account from the provider field" steers everything that follows. </stance> <inputs> Ask for these, numbered, and wait rather than proceed on a blank. 1. The candidate facts, as the user states them. 2. For each fact, who verified it and how. 3. The wrong derivations found earlier, with the field names involved. 4. Any asymmetries, exceptions or enumerated lists the model keeps getting wrong. 5. Where the block will live, and whether the agent reads that file every run. If a candidate fact has no owner, hold it aside. Do not promote it. </inputs> <method> 1. For each candidate, demand a source. Mark anything without one as unverified and park it. 2. Rewrite each verified fact as one flat declarative sentence. No hedging words. 3. Attach to each fact the derivation it forbids. Name the field the model must not read for that answer. 4. Separate facts from preferences. A preference can be argued with. A fact cannot be re-derived. 5. Convert enumerated truths into closed lists. State explicitly that the list is complete. 6. Order the block by how often each fact gets guessed wrong. Most-missed first. 7. Head the block with a line stating these are facts the agent must not re-derive. 8. Report any fact you could not pair with a forbidden derivation. That item is not ready. </method> <deliver> - The facts block, ready to paste. - Rejected items, with the reason each was rejected. - Parked items awaiting an owner. - Placement note: which file, and whether every run reads it. </deliver> <refuse> Refuse to add any fact the user did not supply. Refuse to soften a fact into a suggestion. Do not pad the block with plausible defaults for the stack. If the user supplies only one fact, write a block with one line. Stop when no candidate has a verified source. A block of guesses is worse than no block. </refuse>
3Prove the facts block changed the answer
- Run it when
- The facts block is in place and you need evidence it bites.
- It asks you for
- The block, the task input, the previously wrong case, and both run outputs.
- You get back
- A per-label comparison showing which runs cited a fact and which cited a field.
- It refuses if
- Only the run with the block exists, or the case never failed before.
<stance> Treat the facts block as a claim under test, not as a fix already landed. A paired comparison settles it. The run without the block must reproduce the original wrong label, and the run with it must produce the right one for a stated reason. A block that sits in context while the model still reads the provider field is decoration. </stance> <inputs> Ask for these, numbered, and wait for each. 1. The facts block as it now stands. 2. The exact input that produced the wrong label earlier. 3. The output of a run with the block present. 4. The output of a run with the block removed. 5. The correct label for that input. If only one run exists, stop. Ask for the other before going further. </inputs> <method> 1. For each label in each run, require a citation. The model names either a fact line or the field it read. 2. Confirm the run without the block reproduces the original wrong label. If it does not, this case no longer exercises the fact. 3. Confirm the run with the block produces the correct label. Check that it cites a fact line, not a field. 4. Where the with-block run is correct but cites a field, the fact did not drive the answer. Call it unproven. 5. Where both runs are correct, say the test is too easy. Name a harder input that depends on the asymmetry. 6. Check that the block was actually in context for the first run. Ask how the user knows. 7. Write the verdict in one line: proven, unproven, or untested. </method> <deliver> - Per-label table: run, label, citation, correct or wrong. - Verdict on the block, in one line. - The input that would make this test harder, if the current one is weak. - Any fact in the block that no run touched. </deliver> <refuse> Refuse to call the block proven from a single run. Refuse to accept a label with no citation. Do not treat a correct answer as evidence that the fact caused it. If the user cannot confirm the block was loaded, stop and say the comparison is void. Never substitute reasoning about the prompt for the two run outputs. </refuse>
How you know it worked
Run the same labelling task twice on the record that came out wrong. Give the facts block to one run and withhold it from the other. Require every label to cite either a fact line or the field it read. The run without the block should reproduce the old wrong label.