Passante A Passante publication

En Passante

In Passing
Field Kit

A blocked PTY write froze every tab

Never hold a shared, app-wide lock across an I/O call that can block indefinitely — take the lock to look the handle up, then release it and do the write behind the handle's own lock.

What happened

On 2026-07-24 a task prompt just past ~1KB was sent as one single-quoted argument on one PTY line; the kernel write() never returned, and because the app-wide sessions mutex was held for the whole blocked call, every tab, the resize handler and the background status poller queued behind it and the entire app froze.

2026-07-24 Rust · long-running desktop app with PTY sessions measured: return time of a 1288-char single-line PTY write
4 prompts, in order
  1. Locate the lock held across blocking I/O The whole app is frozen, not just the pane you were typing into, and you have the hung process still running.
  2. Split lookup lock from write lock You have confirmed a shared map lock is held across the blocking write and need to restructure it without introducing new races.
  3. Chunk large writes to a tty A single large payload written in one call can overflow the receiving buffer and block.
  4. Write a test that would have failed The fix is in and you need a regression test that pins the actual property, not a proxy for it.

1Locate the lock held across blocking I/O

Run it when
The whole app is frozen, not just the pane you were typing into, and you have the hung process still running.
It asks you for
A stack sample of the live hung process, the list of every call site that takes the shared session-map lock, and what was sent immediately before the freeze.
You get back
A named lock, the exact blocked syscall the lock is being held across, and the list of unrelated features that queue behind it.
It refuses if
The sample was taken after the process recovered, or no sample exists at all.
copy this into a model
<stance>
You have chased whole-application freezes the expensive way and you know which wrong answer is most persuasive: that this is a deadlock between two locks, or a hung UI thread, or a slow renderer. It usually is not. The common shape is one lock, correctly taken, held across a single call into the kernel that never returns. You do not accept a theory that a live stack sample has not confirmed.
</stance>

<inputs>
Ask for, and wait for, each of these before doing anything else:
1. A stack sample of the hung process taken while it is still hung (on macOS, `sample` against the pid). If the process has already been killed or recovered, stop and say the question cannot be answered until it is reproduced.
2. Every call site that acquires the shared session/handle map lock, with the code that runs while it is held.
3. What the user did in the seconds before the freeze — specifically the size and shape of the last payload written.
</stance>

<method>

1. In the sample, find the thread that is not making progress and read down to the deepest frame. Name the syscall it is sitting in. If that frame is a write, a read, or a poll, the freeze is an I/O block, not a lock cycle — say so explicitly.
2. Walk back up that thread's frames to the point where the shared lock was acquired. State plainly whether the lock is still held at the blocked frame. If the frames do not show this, say the evidence does not settle it and ask for a sample with full frames.
3. List every other thread in the sample that is parked waiting on that same lock. Map each one to a user-visible feature. This list is the blast radius and it is the reason the freeze looked app-wide.
4. Confirm the trigger by size: check whether the last payload crossed a buffer boundary — a tty's canonical-mode input queue will accept a modest line and block on a larger one from the same code path.
5. Enumerate every call site that ever writes through the blocked path. If it is a small number, say the number; that is the scope of the fix.

</method>

<deliver>

- Blocked frame: the exact syscall and thread
- Lock held across it: name and acquisition point
- Blast radius: features parked on the same lock
- Trigger: payload size and the buffer it overflowed
- Write call sites that need the fix

</deliver>

<refuse>
Do not name a root cause without a live stack sample showing the blocked frame. Do not propose raising a timeout, adding a retry, or moving work to another thread as the fix — none of those release the lock. If the sample shows two locks in a cycle, say so and stop: this playbook is for the single-lock-across-blocking-I/O shape and does not apply. If asked to guess based on the symptom description alone, refuse and say what evidence would settle it.
</refuse>

2Split lookup lock from write lock

Run it when
You have confirmed a shared map lock is held across the blocking write and need to restructure it without introducing new races.
It asks you for
The current data structure holding sessions, the type of the writer, and every call site that writes.
You get back
A refactor plan giving each writer its own lock, with the shared lock scoped to the lookup only.
It refuses if
The writer type cannot be cloned or shared out of the map, or the call sites are not fully enumerated.
copy this into a model
<stance>
You have done this refactor before and you know the trap: people 'fix' the freeze by cloning the whole session out of the map, or by switching to a read-write lock, and the blocking write still happens under a guard that something else needs. The only structural fix is that the shared lock's critical section contains a lookup and nothing else. You also know that giving each writer its own lock changes the ordering guarantees for concurrent writes to the same session, and you say so rather than letting the caller discover it.
</stance>

<inputs>
Ask for, and wait for, all of these:
1. The declaration of the shared sessions structure and its lock type.
2. The writer's type and whether it can live behind its own reference-counted mutex.
3. Every call site that writes to a session — including any secondary bridge or external command surface, not just the primary UI path. If the caller cannot say the list is complete, stop and have them grep for the write function first.

</inputs>

<method>

1. Introduce a handle accessor whose entire body is: take the shared lock, look the session up, clone out a reference-counted handle to its writer, drop the shared lock. Show the code. The lock guard must not outlive the function.
2. Move the write itself to operate on the returned handle, taking only the per-writer lock. Confirm by reading the code that no path can hold both locks at once; if one can, say which, and restructure until none can.
3. Check the error paths. A missing session must return an error while the shared lock is scoped to the lookup, not while a write is pending.
4. State the new concurrency semantics out loud: two writes to the same session now serialise on that session's writer lock; writes to different sessions no longer serialise at all. Say whether any caller depended on the old global ordering.
5. Apply the change at every enumerated call site. Name the count. If a call site was found that the caller did not list, stop and report it before continuing.

</method>

<deliver>

- Handle accessor: the code, with the lock scope marked
- Write path: the code, taking only the per-writer lock
- Two-lock audit: paths checked, result
- Semantics changed: ordering guarantees before and after
- Call sites updated: the list and the count

</deliver>

<refuse>
Do not swap the mutex for a read-write lock and call it fixed — a reader guard held across a blocking write blocks writers just as hard. Do not spawn the write onto a background task as the primary fix; that hides the block rather than removing it, and the lock is still held if the guard travels with the task. If the writer type cannot be shared out of the map, stop and say the refactor requires changing that type first. If any write call site is unaccounted for, refuse to declare the fix complete.
</refuse>

3Chunk large writes to a tty

Run it when
A single large payload written in one call can overflow the receiving buffer and block.
It asks you for
The maximum payload size seen in production and the receiving end's mode.
You get back
A bounded chunked-write routine with a pause between chunks, and a statement of what it does not fix.
It refuses if
The receiving end is not a tty or its buffering behaviour is unknown.
copy this into a model
<stance>
You have written a chunking loop that made the freeze go away in testing and reappear on a slower machine, because the chunk size was chosen by feel and nothing paced the writes. You also know the seductive misreading here: chunking makes the write return promptly, and people then claim it made the paste fast. It did not. Those are two different properties and you keep them apart in everything you say.
</stance>

<inputs>
Ask for, and wait for:
1. The largest single payload the system has actually been asked to write, in characters. If unknown, ask for the incident payload rather than assuming.
2. What is on the receiving end and what input mode it is in — a shell in canonical mode has a bounded input queue and will block a writer that exceeds it in one call.
3. Whether writes to this destination must be atomic from any caller's perspective.

</inputs>

<method>

1. Replace the single write of the whole payload with a loop that writes bounded chunks and pauses briefly between them. The pause is what lets the reader drain the queue; without it the loop can still fill the buffer in one burst.
2. Justify the chunk bound against the receiving buffer, not against a round number. Say explicitly if the receiver's buffer size is unknown and you are choosing conservatively.
3. Flush per chunk, and propagate a write error from any chunk immediately rather than after the loop.
4. Confirm the chunking sits behind the per-writer lock, not the shared one, so a long payload delays only its own session.
5. Separate the two properties in writing: (a) the write call now returns promptly; (b) the receiver may still take much longer to render or process the payload. For an interactive shell, a paste wider than one terminal width is legitimately redrawn character-by-character by the shell's own line editor, and that cost scales with the shell's prompt and theme, not with your write path. Name bracketed paste mode as the correct fix for (b) and scope it as separate work — it can change how control bytes such as Ctrl-C are delivered and needs cross-shell verification before shipping.

</method>

<deliver>

- Chunked write routine: the code
- Chunk bound: the number and the reason for it
- Error handling per chunk
- What this fixes: write returns promptly
- What this does not fix: receiver-side rendering, and the follow-up that would

</deliver>

<refuse>
Do not present chunking as a fix for slow echo-back or slow rendering; say plainly that it is not. Do not ship bracketed paste mode in the same change as an incident fix — it changes control-byte delivery and needs cross-shell verification first. If the receiving end's buffering behaviour is unknown and cannot be determined, stop and say the chunk bound cannot be justified yet.
</refuse>

4Write a test that would have failed

Run it when
The fix is in and you need a regression test that pins the actual property, not a proxy for it.
It asks you for
The exact incident payload length and access to a real session in tests.
You get back
A test that writes an incident-sized line to a real shell and asserts the call returns inside a deadline.
It refuses if
The test passes against the pre-fix code, or it asserts on echo-back rather than on write return.
copy this into a model
<stance>
You have shipped regression tests that passed against the broken code and told nobody anything. You know the two ways it happens here: the test uses a mock writer that cannot block, or it asserts on something downstream — the echoed text coming back — which is slower, noisier, and controlled by things the fix does not touch. The property that was violated was 'the write call returns'. That is the only thing the test asserts.
</stance>

<inputs>
Ask for, and wait for:
1. The exact character length of the payload from the real incident. Use that number; do not round it.
2. Whether the test harness can open a real interactive shell session, or only a mock. If only a mock, stop — a mock writer cannot reproduce a blocked kernel write.
3. The deadline the team is willing to call 'promptly'.

</inputs>

<method>

1. Build the payload as a single line of exactly the incident length — one line, no embedded newlines, since the single-line shape is what filled the canonical-mode queue.
2. Open a real shell session in the test. Write the line through the same production code path the app uses, not a test-only shim.
3. Assert only that the write call returns within the deadline. Do not read back, do not match on echoed output, do not wait for a prompt to redraw.
4. Write a comment in the test explaining why it stops there: the shell's own line editor redraws a paste exceeding one terminal width character-by-character, which is real, much slower, and scales with the shell's prompt and theme complexity — separate from what this test guards.
5. Falsify the test. Revert the fix — put the write back under the shared lock, or restore the single unchunked write — and run it. If it still passes, the test is not exercising the property; fix the test before trusting it. Report the result of this step either way.
6. Add a second observable if the harness allows: while a large write is in flight, issue a command that needs the shared lock and assert it answers. That pins the blast-radius half of the bug.

</method>

<deliver>

- Test: the code, with the payload length and deadline as literals
- Comment explaining the deliberate stopping point
- Falsification result: pass or fail against the reverted fix
- Optional second assertion: unrelated command answers during a write

</deliver>

<refuse>
Do not assert on echoed output; it makes the test slow and flaky for reasons the fix does not control. Do not use a mock writer — say plainly that a mock cannot block and therefore cannot regress. If the test passes against the pre-fix code, refuse to call it a regression test and say what it is actually measuring. If the incident payload length is unknown, stop and ask for it rather than picking a round number.
</refuse>
How you know it worked

Run the regression test that writes a 1288-character single line to a real zsh session and asserts the write returns within 5s — then, while a large write is in flight, ask the app for its tab list and confirm it answers. Put the writer back behind the shared sessions lock and the same test should stop returning promptly; if it passes either way, the test is not exercising the property.

from a real incident, 2026-07-24 · stack Rust · long-running desktop app with PTY sessions · verified by return time of a 1288-char single-line PTY write · concurrencyptydeadlockregression-tests

← every Field Kit