Field Kit

Background indexing still owns the JavaScript thread

Measure foreground responsiveness while background work runs, using an observer that cannot freeze on the same thread.

What happened

An artifact index blocked requests while reading and projecting a real library; a separate-process responsiveness test recorded a slowest reply of 48 ms after time slicing, while the old behavior exceeded the test's response deadline.

2026-09-27 Node.js · SQLite · background indexing measured: Historical separate-process HTTP latency under indexing

2 prompts, in order

Locate the uninterrupted work A server stops answering during a background indexing pass.
Measure foreground work during the crawl The indexing loop now yields and needs proof of both responsiveness and correctness.

1Locate the uninterrupted work

Run it when: A server stops answering during a background indexing pass.
It asks you for: The actual indexing stages, database read shapes, request latency logs and representative library size.
You get back: The longest uninterrupted operations and the smallest changes that let requests run between them.
It refuses if: Do not equate async function syntax with work that yields to incoming requests.

copy this into a model

<stance>
Your job is to separate an observed failure from a convincing explanation that has never been checked. Keep the reader's actual system in view. A result is useful when another person can repeat the observation and get a different answer if the proposed repair is wrong.
</stance>

<inputs>
Ask for these before drawing a conclusion. Wait for anything essential that is missing.
1. The actual indexing stages, database read shapes, request latency logs and representative library size.
2. The exact user-visible symptom, including the original output and the time it was observed.
3. The implementation and launch conditions responsible for that output. Read the relevant source in full.
4. What may be changed safely, and which actions require the owner's approval.
5. Any existing verification that could distinguish the proposed cause from the alternatives.
</inputs>

<method>
1. Trace the indexing pass from discovery through import, projection, search refresh and serialization. Identify synchronous operations and whole-table reads inside each stage, including repeated work hidden in helper calls.
2. Measure uninterrupted time independently of total indexing time. A stage can take a reasonable total duration while still holding the single JavaScript thread long enough to exceed every client deadline.
3. Inspect lookups inside loops for repeated full scans. Reuse a row already in hand when that preserves behavior; separate this improvement from yielding between units of work.
4. Propose bounded pages and time slices at natural transaction and iteration boundaries. Preserve atomicity, ordering and output format. State which expensive primitive can still exceed the slice budget so the plan remains checkable.
</method>

<deliver>
- OBSERVED: the behavior actually established, with the output that supports it.
- EXPLANATION: what the observation settles and what remains unknown.
- NEXT ACTION: one bounded action, its owner, and its expected result.
- PROOF: the observation that would establish success, plus the result that would disprove it.
</deliver>

<refuse>
Do not equate async function syntax with work that yields to incoming requests.
Do not invent measurements, defaults, source contents, or a successful run. Keep historical results separate from today's observations. Never transmit private inputs or perform a paid or external action just to complete this check. If access or authorization is missing, stop and name the exact missing condition.
</refuse>

2Measure foreground work during the crawl

Run it when: The indexing loop now yields and needs proof of both responsiveness and correctness.
It asks you for: The real index entry point, expected resulting row count, latency budget and independent request observer.
You get back: Worst request latency during the full pass, request coverage and resulting index correctness.
It refuses if: Do not observe the server solely from a client running on the same blocked event loop.

copy this into a model

<inputs>
Ask for these before drawing a conclusion. Wait for anything essential that is missing.
1. The real index entry point, expected resulting row count, latency budget and independent request observer.
2. The exact user-visible symptom, including the original output and the time it was observed.
3. The implementation and launch conditions responsible for that output. Read the relevant source in full.
4. What may be changed safely, and which actions require the owner's approval.
5. Any existing verification that could distinguish the proposed cause from the alternatives.
</inputs>

<method>
1. Use the existing isolated indexing test or an approved copy of the real library. Keep production data untouched. Establish the request observer before indexing starts and keep it running through the whole pass.
2. Send requests from a separate process so a stall cannot freeze both the server and its measuring client. Capture worst latency and the number of observations across the pass; an average can hide the harmful pause.
3. Verify indexing actually performed the expected work and preserved every expected result. An optimization that skips discovery or drops rows could produce excellent latency while failing the product's job.
4. Compare against the previous implementation using the same isolated workload when the existing test supports it. Treat that comparison as evidence about this workload and machine, with no universal performance claim.
</method>

<refuse>
Do not observe the server solely from a client running on the same blocked event loop.
Do not invent measurements, defaults, source contents, or a successful run. Keep historical results separate from today's observations. Never transmit private inputs or perform a paid or external action just to complete this check. If access or authorization is missing, stop and name the exact missing condition.
</refuse>

How you know it worked

Run the real indexing operation while a separate process sends requests throughout it. Verify the index contains the expected rows and the worst request meets the chosen budget. A fast response before indexing starts and a completed index alone are both insufficient.

← every Field Kit