Field Kit

The health probe can be the broken component

A failed probe describes the model only after the probe can perform the operation that model supports.

What happened

A model-health daemon had no installed scheduler and lacked a required client dependency; its local probes also used registry keys instead of runtime model tags, sent generation requests to embedding models and tried API checks against subscription-only CLI models.

2026-07-07 Python · launchd · local model registry measured: Scheduler, dependencies, model identifiers and workload-specific probes

2 prompts, in order

Validate the observer first A model registry reports broken providers or stops producing fresh health reports.
Check the repaired probe boundaries The scheduler and workload-specific probe selection have been repaired.

1Validate the observer first

Run it when: A model registry reports broken providers or stops producing fresh health reports.
It asks you for: The scheduler definition, last-run metadata, execution environment, model identifiers and probe implementation.
You get back: An account of observer failures, unsupported probes and genuine provider failures.
It refuses if: Do not retire a model because an invalid probe failed.

copy this into a model

<stance>
Your job is to separate an observed failure from a convincing explanation that has never been checked. Keep the reader's actual system in view. A result is useful when another person can repeat the observation and get a different answer if the proposed repair is wrong.
</stance>

<inputs>
Ask for these before drawing a conclusion. Wait for anything essential that is missing.
1. The scheduler definition, last-run metadata, execution environment, model identifiers and probe implementation.
2. The exact user-visible symptom, including the original output and the time it was observed.
3. The implementation and launch conditions responsible for that output. Read the relevant source in full.
4. What may be changed safely, and which actions require the owner's approval.
5. Any existing verification that could distinguish the proposed cause from the alternatives.
</inputs>

<method>
1. Establish whether the scheduled process is installed and when it actually last ran. Read the scheduler's interval and launch behavior from its definition; a reassuring installation sentence cannot override those values.
2. Inspect the exact interpreter and dependencies used by that process. A client installed in an interactive environment can still be absent from the daemon's environment. Diagnose import failure before interpreting it as provider unavailability.
3. Compare registry identifiers with the runtime model tag sent to the local service. Match the endpoint to the workload: embedding models and text-generation models do not accept interchangeable requests.
4. Identify subscription-only CLI entries and missing-key behavior. Keep excluded or unsupported checks outside failure-based retirement decisions. List the unobserved models by name without presenting them as either healthy or broken.
</method>

<deliver>
- OBSERVED: the behavior actually established, with the output that supports it.
- EXPLANATION: what the observation settles and what remains unknown.
- NEXT ACTION: one bounded action, its owner, and its expected result.
- PROOF: the observation that would establish success, plus the result that would disprove it.
</deliver>

<refuse>
Do not retire a model because an invalid probe failed.
Do not invent measurements, defaults, source contents, or a successful run. Keep historical results separate from today's observations. Never transmit private inputs or perform a paid or external action just to complete this check. If access or authorization is missing, stop and name the exact missing condition.
</refuse>

2Check the repaired probe boundaries

Run it when: The scheduler and workload-specific probe selection have been repaired.
It asks you for: The installed scheduler metadata, existing probe-selection checks, child-process environment rules and report format.
You get back: A valid observation path plus named skips and explicit limits on live-provider verification.
It refuses if: Do not run billable provider probes or install a scheduler without the owner's authorization.

copy this into a model

<inputs>
Ask for these before drawing a conclusion. Wait for anything essential that is missing.
1. The installed scheduler metadata, existing probe-selection checks, child-process environment rules and report format.
2. The exact user-visible symptom, including the original output and the time it was observed.
3. The implementation and launch conditions responsible for that output. Read the relevant source in full.
4. What may be changed safely, and which actions require the owner's approval.
5. Any existing verification that could distinguish the proposed cause from the alternatives.
</inputs>

<method>
1. Use the existing isolated checks to verify CLI-only entries are excluded and local model identifiers reach the correct operation. Keep generation and embedding selection as separate outcomes so one cannot conceal the other.
2. Inspect credential injection into the scheduled child. Keys should arrive through the existing approved broker rather than being copied into the durable scheduler definition. Review names and control flow without printing values.
3. Compare configured timing with the scheduler's read-only registration metadata. Confirm the report exposes observation time and skipped models. A loaded process with an old report still leaves current provider health unknown.
4. Separate inspected selection behavior from an actual scheduled run and external-provider results. Report each unperformed observation explicitly. Any future paid check needs its own approved scope and must not be smuggled into a routine daemon-verification step.
</method>

<refuse>
Do not run billable provider probes or install a scheduler without the owner's authorization.
Do not invent measurements, defaults, source contents, or a successful run. Keep historical results separate from today's observations. Never transmit private inputs or perform a paid or external action just to complete this check. If access or authorization is missing, stop and name the exact missing condition.
</refuse>

How you know it worked

Validate the observer first. Verify that the scheduler exists, its environment can import the client and each model receives a supported probe. Skipped models must stay distinct from failed ones. Read the configured schedule itself. Registering the daemon does not establish that it ran successfully or that every provider is healthy.

← every Field Kit