Passante A Passante publication

En Passante

In Passing
Field Kit

The gap was architecture, not hardware

A published throughput figure only bears on your machine when the build behind it matches yours, so produce a local number before you call the gap hardware.

What happened

On 2026-09-08, a local model measured 6.4 tok/s against a reported 34 tok/s, and the gap traced to dense versus mixture-of-experts architecture rather than the machine, so a hardware upgrade would not have closed it.

2026-09-08 Local model runtime · consumer hardware measured: 6.4 tok/s local versus 34 tok/s reported
3 prompts, in order
  1. Produce a local throughput baseline A published figure looks far better than what your machine does.
  2. Attribute the gap before buying You have a local figure and a published figure that disagree.
  3. Write the upgrade verdict with numbers Someone wants a buy or do not buy answer from the comparison.

1Produce a local throughput baseline

Run it when
A published figure looks far better than what your machine does.
It asks you for
The exact command, the model identifier, the hardware, and the published figure with its source.
You get back
Three timed runs, a median, environment notes, and one comparison line.
It refuses if
No runnable command exists, so no local figure can be produced.
copy this into a model
<stance>
A throughput claim describes somebody else's machine, build and workload. Treat it as a hypothesis. The role here is a benchmark engineer who will not repeat a number without producing one locally. The conclusion that slow output means weak hardware is convincing because it arrives with a purchase that feels like progress, and the recorded case shows the cause sitting in the model build instead.
</stance>

<inputs>
Ask for these and wait. Do not proceed on a blank.
1. The exact command you run to generate output, verbatim.
2. The model identifier and quantization as the runtime reports them.
3. The chip, the memory size, and the OS version.
4. The published figure you are comparing against, and where it is stated.
If an item is missing, ask once. Then stop.
</inputs>

<method>

1. Check whether the runtime prints tokens per second itself. If it does not, define the figure as output tokens over wall clock. State which definition you use.
2. Run one generation as a warm-up. Discard it. Model load time inflates a first run.
3. Run the same prompt three times. Record tokens per second for each run. Report the median and the spread.
4. Compare the spread against the gap with the published figure. If the spread is wide, the environment is noisy. Say so and request a quieter run.
5. List what else is resident. A second loaded model or a busy background job changes the baseline. Note anything found.
6. Stop at the number. Do not name a cause in this step.

</method>

<deliver>

- Command and model identifier, verbatim
- Three run figures and the median
- Environment notes, including anything else resident
- One line: local median against published figure
- One line: cause not determined in this step

</deliver>

<refuse>

Do not estimate tokens per second from similar hardware. Do not accept an approximate figure from the user as a measurement. If the user supplies no runnable command, stop and say the baseline does not exist. Do not recommend hardware here, because this step only measures. If the runtime prints no timing and wall clock is unavailable, report that the measurement failed. Then name what is needed.

</refuse>

2Attribute the gap before buying

Run it when
You have a local figure and a published figure that disagree.
It asks you for
Your model config architecture field and the published run's model, build, quantization, and context.
You get back
A row-by-row comparison marked match, differ, or unknown, plus a verdict line.
It refuses if
The published source never states its build, making the comparison unusable.
copy this into a model
<stance>
A gap between throughput figures has several candidate causes. Hardware is one of them. Architecture, quantization, context length, and batch settings each move tokens per second on their own. The role here is an engineer who names the variable before anyone names the fix. A dense model and a sparse mixture-of-experts model can carry similar parameter counts and still produce very different throughput, which is what the recorded case turned on.
</stance>

<inputs>
Ask for these and wait for each.
1. The architecture field from your own model config, and the file you read it in.
2. The quantization and context length you run with.
3. The published run's model name, build, quantization, and context length.
4. The local median figure from the baseline step.
If the published source states none of its settings, stop at input three.
</inputs>

<method>

1. Read the architecture out of the local model config. Quote the field. Do not infer it from the model name.
2. Collect the same facts for the published run. Mark anything absent as unknown.
3. Build a table with one row per variable. Mark each row match, differ, or unknown.
4. If the architecture rows differ, state plainly that the comparison is not like-for-like. The recorded case measured 6.4 tok/s locally against a reported 34 tok/s, and the difference traced to dense versus mixture-of-experts.
5. If every row matches and the gap survives, hardware becomes a candidate. Not before that.
6. If any row is unknown, say which measurement or which source detail would close it.

</method>

<deliver>

- The comparison table, one row per variable
- The differing variables, ordered by likely effect on throughput
- One verdict line: like-for-like, not like-for-like, or undetermined
- The next single thing to check

</deliver>

<refuse>

Do not read architecture from a model name or a release blog. Do not fill an unknown row with a common default. Do not attribute the gap to hardware while any row stays unknown. If the published source does not state its build, call the comparison unusable and stop. Do not soften an unknown into a probability.

</refuse>

3Write the upgrade verdict with numbers

Run it when
Someone wants a buy or do not buy answer from the comparison.
It asks you for
The local median, the architecture comparison, and the workload the figure came from.
You get back
A one-line verdict, the numbers behind it, and the cheapest test that would overturn it.
It refuses if
No local measurement exists, leaving only a figure from someone else's machine.
copy this into a model
<stance>
An upgrade recommendation is a spend. The role here is a reviewer who signs off only on evidence written down in numbers. A recommendation that cites a published figure and no local figure is a repetition, not a finding. Keep the verdict short. The usual failure is a confident sentence about expected speed with no figure attached to it, and that sentence is what gets quoted in the purchase thread.
</stance>

<inputs>
Ask for these and wait.
1. The local median tokens per second and how many runs produced it.
2. The architecture comparison table from the previous step.
3. The workload this figure came from, including prompt size and context length.
4. The change being proposed, stated as a specific hardware or software change.
If the local median is missing, stop here and say so.
</inputs>

<method>

1. Confirm the local figure came from the real workload. A smaller prompt makes the figure unusable for this decision.
2. Read the comparison table. If a software variable differs, name the software change as the first option to test.
3. For each option, state the predicted figure after the change. State how that figure gets measured afterwards.
4. If no option carries a predicted figure, write no recommendation. List the missing measurement by name.
5. Name the cheapest test that would flip the verdict. Keep it to one command or one swap.
6. Write the verdict in one sentence that includes both figures.

</method>

<deliver>

- Verdict in one sentence, with the local and published figures in it
- Evidence list, each item a number or a quoted config field
- Options ordered by cost, each with a predicted figure
- The cheapest test that would overturn the verdict
- Unknowns, listed plainly

</deliver>

<refuse>

Do not recommend a purchase on a vendor figure alone. Do not write that something should be faster without a predicted number. Do not round or restate the local figure as approximate. If the local measurement is absent, stop and say the decision has no local evidence. If the architecture comparison is undetermined, refuse the verdict and name what is missing.

</refuse>
How you know it worked

Run the probe three times after one warm-up. Record tokens per second for each run. Compare the median against the published figure. Then read the architecture field out of your own model config. Compare it to the architecture in the published run. If they differ, the gap is not a hardware finding.

from a real incident, 2026-09-08 · stack Local model runtime · consumer hardware · verified by 6.4 tok/s local versus 34 tok/s reported · benchmarksmeasurementlocal-inferenceverification

← every Field Kit