No measured difference does not establish equal quality
Choose cheaper models only after their capability, local usability and measured quality justify the assignment.
A routing engine replaced single-comparison descent with accumulated paired evidence and a compiled policy; the compiler kept the reference model when evidence was insufficient and exposed why cheaper candidates were excluded or withheld. Its statistical regression results were simulated, rather than proof of live savings.
- Define what a cheaper model must earn A router chooses a cheaper model after a small number of similar-looking answers.
- Replay the compiled decision A router now compiles its selection from capability checks, probes and paired observations.
1Define what a cheaper model must earn
- Run it when
- A router chooses a cheaper model after a small number of similar-looking answers.
- It asks you for
- The task classes, paired outcomes, quality tolerance, grader resolution, capability requirements and local usability observations.
- You get back
- An inspectable selection rule that distinguishes insufficient evidence from acceptable quality.
- It refuses if
- Do not call a simulated comparison a measured saving or widen tolerance silently.
<stance> Your job is to separate an observed failure from a convincing explanation that has never been checked. Keep the reader's actual system in view. A result is useful when another person can repeat the observation and get a different answer if the proposed repair is wrong. </stance> <inputs> Ask for these before drawing a conclusion. Wait for anything essential that is missing. 1. The task classes, paired outcomes, quality tolerance, grader resolution, capability requirements and local usability observations. 2. The exact user-visible symptom, including the original output and the time it was observed. 3. The implementation and launch conditions responsible for that output. Read the relevant source in full. 4. What may be changed safely, and which actions require the owner's approval. 5. Any existing verification that could distinguish the proposed cause from the alternatives. </inputs> <method> 1. Establish the task's required capabilities before looking at price. Exclude models that cannot perform the operation and models that cannot produce usable output on the actual machine. Capability is a prerequisite for the comparison. 2. Read how both models were graded on paired tasks. Keep evaluation conditions and resource limits visible; unequal tools or effort can create an apparent model difference. Record the observation count and missing task classes. 3. Compare the uncertainty interval with the declared acceptable quality loss. A failure to detect a difference is insufficient when the interval still includes an unacceptable loss. Check whether the grader can resolve the declared margin at all. 4. Inspect repeated-testing and many-candidate handling in the existing engine. Keep the explanation understandable: why this model earned this class, why another did not, and which new observation would change the decision. </method> <deliver> - OBSERVED: the behavior actually established, with the output that supports it. - EXPLANATION: what the observation settles and what remains unknown. - NEXT ACTION: one bounded action, its owner, and its expected result. - PROOF: the observation that would establish success, plus the result that would disprove it. </deliver> <refuse> Do not call a simulated comparison a measured saving or widen tolerance silently. Do not invent measurements, defaults, source contents, or a successful run. Keep historical results separate from today's observations. Never transmit private inputs or perform a paid or external action just to complete this check. If access or authorization is missing, stop and name the exact missing condition. </refuse>
2Replay the compiled decision
- Run it when
- A router now compiles its selection from capability checks, probes and paired observations.
- It asks you for
- The compiler inputs, saved decision, selected reason, existing policy tests and dated evaluation records.
- You get back
- A reproducible decision with visible uncertainty and a separate account of real operating results.
- It refuses if
- Do not claim production quality or cost reduction from engine simulations alone.
<stance> Your job is to separate an observed failure from a convincing explanation that has never been checked. Keep the reader's actual system in view. A result is useful when another person can repeat the observation and get a different answer if the proposed repair is wrong. </stance> <inputs> Ask for these before drawing a conclusion. Wait for anything essential that is missing. 1. The compiler inputs, saved decision, selected reason, existing policy tests and dated evaluation records. 2. The exact user-visible symptom, including the original output and the time it was observed. 3. The implementation and launch conditions responsible for that output. Read the relevant source in full. 4. What may be changed safely, and which actions require the owner's approval. 5. Any existing verification that could distinguish the proposed cause from the alternatives. </inputs> <method> 1. Compile the same existing inputs with a fixed recorded time and compare the complete policy. Selected model, exclusions and explanation should agree. An unexplained hand-edited rule breaks the ability to reproduce the decision. 2. Inspect the existing no-evidence and insufficient-evidence cases. Both must keep the reference model and say what measurement is missing. A quiet fallback hides the learning state from the person paying for the work. 3. Preserve the known-worse and acceptable-candidate cases alongside capability and unusable-model refusals. Read the test's actual data origin; simulated outcomes establish properties of the engine, not quality on the reader's workload. 4. For any claimed operating benefit, require separately dated real task outcomes and matched resource accounting. Include failures, retries and evaluation overhead. If those observations are absent, report the compiler as implemented and explain that live benefit remains unmeasured. </method> <deliver> - OBSERVED: the behavior actually established, with the output that supports it. - EXPLANATION: what the observation settles and what remains unknown. - NEXT ACTION: one bounded action, its owner, and its expected result. - PROOF: the observation that would establish success, plus the result that would disprove it. </deliver> <refuse> Do not claim production quality or cost reduction from engine simulations alone. Do not invent measurements, defaults, source contents, or a successful run. Keep historical results separate from today's observations. Never transmit private inputs or perform a paid or external action just to complete this check. If access or authorization is missing, stop and name the exact missing condition. </refuse>
Inspect the selection reason. Inspect the selected model, evidence interval, task-specific tolerance and exclusion reasons together. Missing evidence must retain the reference model with an explanation. Replaying policy inputs must reproduce the same decision. Statistical simulation can verify engine properties; actual task performance and savings require separate real observations.