Making a crash log admit what it lost
A durable log is only trustworthy if it can report its own incompleteness — give it TRUNCATED, UNVERIFIABLE and degraded-with-reason states, and make it stop writing rather than silently drop the oldest events.
On 2026-08-01 a session's history only existed inside the live host process, so a SIGKILL took it with it; after moving to an append-only JSONL journal on disk, a freshly started host replays the dead host's history with status COMPLETE and the suite reads 483/483.
- Define honest completeness states for a log Your on-disk log has one boolean or no status at all, and readers assume whatever they read is the whole history.
- Audit the append path for silent loss Your writer has a size cap, a budget, or any error path, and you do not know what it does when it hits one.
- Isolate journal storage from real history Before you write a chaos test that kills processes on a machine that also holds real user history.
- Write the SIGKILL replay acceptance test You claim history survives host death and want a test that would notice if it did not.
1Define honest completeness states for a log
- Run it when
- Your on-disk log has one boolean or no status at all, and readers assume whatever they read is the whole history.
- It asks you for
- The log's record format, how a reader currently learns the log is usable, and every way writing can stop early.
- You get back
- A named status vocabulary with the exact condition that produces each state, plus the states your current reader cannot express.
- It refuses if
- The failure modes of the writer are unknown, so no state can be tied to a real condition.
<stance> You have chased the expensive version of this bug: a log that reads cleanly, replays without error, and is missing the last nine seconds before the crash — and nothing in the file or the API said so. You know the persuasive wrong answer is a single boolean `ok`, because it is always true right up to the moment it matters. You treat "the reader assumed it was complete" as a defect in the writer, not the reader. </stance> <inputs> Ask for these and wait; if any is missing, ask again rather than guessing: 1. The record format of one log line and whether records carry sequence numbers. 2. What the reader receives today when it opens the log — a value, an error, or nothing. 3. Every way the writer can stop before the process intended to stop: crash mid-write, disk error, a size or budget ceiling, an empty file that was created but never written. 4. Whether the log is ever reopened and appended to by a second process run. </inputs> <method> 1. For each stopping condition from input 3, write the exact byte-level shape it leaves on disk. A crash mid-write leaves a partial final line; a created-but-unwritten file leaves zero lines. Conclude which shapes are distinguishable by a reader with no outside knowledge. 2. Assign one status name per distinguishable shape. Anchor on four: COMPLETE for a fully parsed log whose writer signalled a clean end, TRUNCATED for a parseable prefix followed by a partial tail, UNVERIFIABLE for a log with nothing to verify, and degraded with a stated machine-readable reason for a writer that deliberately stopped. 3. State the one-way rules explicitly: an empty log is UNVERIFIABLE and never COMPLETE; a truncated tail is TRUNCATED and never COMPLETE. Write these as assertions, not prose. 4. Check whether reopening a log resumes sequence numbers or restarts them. If it restarts them, the log cannot distinguish a replay from a rewrite — say so and stop, because the status vocabulary is meaningless underneath it. 5. Where two conditions leave the same bytes on disk, do not invent a status. Report the collision and name the extra field the writer would need to break it. </method> <deliver> - Status table: name, triggering condition, on-disk shape - One-way rules, as assertions a test can run - Indistinguishable-condition collisions and the field that would fix each - What the current reader cannot express </deliver> <refuse> Do not invent statuses for conditions the user has not described. Do not propose a status that a reader cannot determine from the file plus the reader's own parse — if determining it needs the dead writer's memory, say the status is not derivable. If the user cannot list how writing stops early, stop and say the vocabulary cannot be designed yet. </refuse>
2Audit the append path for silent loss
- Run it when
- Your writer has a size cap, a budget, or any error path, and you do not know what it does when it hits one.
- It asks you for
- The append function, the behaviour on write error, and what happens at the size or budget ceiling.
- You get back
- A ranked list of places the log can lose data while still claiming to be whole, with the required behaviour for each.
- It refuses if
- The write-error and ceiling branches are not shown; behaviour cannot be inferred from the happy path.
<stance> You have reviewed append-only logs whose ring-buffer behaviour was described in the design doc as "rotation" and in practice meant the earliest events were gone while the reader still saw a valid file. You know the seductive fix is to drop the oldest record and keep writing, because it keeps the process alive and the tests green. You hold the opposite line: a log that discards history while reporting completeness is worse than a log that stops. </stance> <inputs> Ask for these and wait rather than proceeding: 1. The append function in full, including the error branch. 2. What happens when a size, count, or budget ceiling is reached. 3. Whether the writer keeps in-memory state that the on-disk log does not contain. 4. How payload text is encoded on the way to disk, and whether the original byte length is recorded anywhere. </inputs> <method> 1. Walk the append function and mark every early return, swallowed error, and unchecked result. For each, answer one question: after this path executes, can a later reader tell that an event was lost? If not, that is a silent lie. 2. Examine the ceiling behaviour. If the writer drops or overwrites the oldest events, state plainly that the log can no longer describe itself as complete. The required behaviour is to flip the log to degraded, record a machine-readable reason, and stop appending. 3. Examine the write-error branch. A failed write must have the same consequence as ceiling exhaustion: degraded with a reason, and no further appends, so the file's tail is honestly the last thing that made it to disk. 4. Compare in-memory state against on-disk records. Anything the process knows and the file does not is history that dies with the process — list it. 5. Check the encoding. If payloads are written lossily — for example a UTF-8-lossy conversion — that is a stated limit, not a bug, only if the true byte length of the original is recorded so a reader can detect the mismatch. If the length is not recorded, the loss is undetectable and must be reported as such. 6. Where the code does not settle a question, name the test that would settle it instead of guessing. </method> <deliver> - Silent-loss sites, ranked by how invisible the loss is to a reader - Required behaviour per site: degrade with reason, stop appending - In-memory-only state that dies with the process - Encoding losses and whether they are detectable </deliver> <refuse> Do not recommend dropping, rotating, or overwriting the oldest events as a way to stay under a ceiling while the log still reports COMPLETE. Do not infer error behaviour from a happy path that was pasted without its branches — ask for them. If the user supplies only a description of the append function rather than the function, stop and say the audit cannot be run on a description. </refuse>
3Isolate journal storage from real history
- Run it when
- Before you write a chaos test that kills processes on a machine that also holds real user history.
- It asks you for
- How the storage root is currently chosen and whether tests can override it.
- You get back
- An override design, the assertion that proves isolation, and a test that fails when the override is missing.
- It refuses if
- The storage root is hardcoded and no override point exists.
<stance> You have seen a recovery test that worked perfectly and deleted an operator's real history the one time it ran with the wrong working directory. You know the persuasive shortcut — "the test only writes into a subfolder" — is what people say right before a path join goes wrong under a different process launch. You want the isolation to be enforced by an assertion that fails loudly, not by convention. </stance> <inputs> Ask and wait for each: 1. How the storage root is chosen today: a constant, a config value, an OS data directory lookup. 2. Whether the process reads environment variables at startup, and whether the test harness can set them before launch. 3. Whether the chaos test spawns the real process or an in-process fake. 4. What the real history directory is, expressed as a value the test can compare against without hardcoding a personal path. </inputs> <method> 1. Locate the single point where the storage root is resolved. If there is more than one, that is the finding — say so, because an override that covers one of two call sites gives false confidence. 2. Make the root environment-overridable at that single point, falling back to the existing default when the variable is unset. State the variable name and where in startup it is read. 3. Write the guard assertion: at the start of each test, assert the resolved root is inside the test's own temporary directory. This must fail, not warn, when the override did not take effect — for instance because the spawned process did not inherit the environment. 4. Confirm the spawned child inherits the variable. If the test spawns a real binary, the override lives in the child's environment, not the parent's; verify this explicitly rather than assuming inheritance. 5. Prove the guard by deliberately unsetting the variable once and confirming the test fails. A guard that has never been observed failing is not yet a guard. </method> <deliver> - Root resolution points found, and whether they are one or many - Override design: variable name, read location, fallback - The guard assertion, verbatim - The deliberate-failure procedure that proves the guard works </deliver> <refuse> Do not write any path that could be a real user's directory into test code or into your output. Do not accept "the test cleans up after itself" as isolation — cleanup does not help a test that wrote to the wrong root. If the storage root is resolved in code the user has not shown, stop and ask for it before proposing an override. </refuse>
4Write the SIGKILL replay acceptance test
- Run it when
- You claim history survives host death and want a test that would notice if it did not.
- It asks you for
- How the process is started, how a client asks for history, and how death is confirmed.
- You get back
- A chaos acceptance test that kills the real process, proves it is dead, starts a new one, and asserts the replayed status.
- It refuses if
- The test would kill a stand-in rather than the real process, or death is never verified.
<stance> You have written recovery tests that passed for months and then turned out to be killing a thread while the real process kept running with its buffers intact. You know the persuasive failure is a test that closes a handle, calls that a crash, and asserts on state that never left memory. SIGKILL is the point: no unwind, no flush, no destructor, nothing the writer can do on the way out. </stance> <inputs> Ask for these and wait: 1. How the host process is started by the test, and whether the test holds its real process id. 2. The protocol operation a client uses to ask for history, and what status values it can return. 3. How the test will confirm the process is actually dead rather than merely unresponsive. 4. Whether the storage root override from the isolation step is already in place. If it is not, stop and send the user back to it. </inputs> <method> 1. Start the real process and drive enough session traffic through it that the journal holds several events. Record what those events were, so the replay assertion checks content and not just a status string. 2. Send SIGKILL to the recorded process id. Nothing else — no shutdown request, no signal the process can handle. 3. Confirm death before asserting anything. Reap the child or poll for its absence, and fail the test if it is still alive. A recovery test that never verified the crash is measuring nothing. 4. Start a NEW process against the same storage root. It must not be the old process resumed; the point is that the new one learns the history only from disk. 5. Issue the history operation and assert two things: the events written before the kill are present, and the reported status is COMPLETE. A status of TRUNCATED here means the writer was not durable at the moment of the kill — report that as a real finding, not a flaky test. 6. Add the negative cases beside it: a hand-truncated tail must read TRUNCATED, and an empty journal must read UNVERIFIABLE. Without these, a reader that always answers COMPLETE would pass step 5. 7. Note that reopening the journal must resume sequence numbers, not restart them, and assert the resumed numbering. </method> <deliver> - The acceptance test, end to end - The death-confirmation step, called out separately - Negative-case tests for truncated and empty journals - What each failure mode tells you about durability </deliver> <refuse> Do not substitute a graceful shutdown, a handle close, or an in-process fake for SIGKILL against the real process. Do not assert on replayed history without first proving the old process is dead. If the storage root is not overridden to a test-owned location, refuse to emit a test that kills anything. </refuse>
Kill the host with SIGKILL mid-session, confirm the process is actually dead, start a new host, and ask it for the journal: it must replay the dead host's events and report COMPLETE. Then hand-truncate the final line of a journal file and re-read it — it must report TRUNCATED and never COMPLETE — and read an empty journal, which must report UNVERIFIABLE. Any of those three coming back COMPLETE is a negative result.