Passante A Passante publication

En Passante

In Passing
Field Kit

Dev server OOM: V8 ceiling, not machine RAM

An out-of-memory crash on a machine with free RAM is a runtime ceiling problem, not a capacity problem — prove which one you have before you buy hardware or start hunting leaks.

What happened

On 2026-08-22 a long-running dev server died with a node::OOMErrorHandler crash mid-way through a screen-thumbnail capture run while rendering one of the heaviest pages (18.5s render, ~20 tab modules plus loaders) on a machine with 48GB of RAM and a post-restart process RSS of only ~520MB.

2026-08-22 Node · a long-running dev server measured: heap ceiling before and after, via v8.getHeapStatistics()
3 prompts, in order
  1. Separate heap ceiling from real memory pressure A long-running dev server just crashed out-of-memory and the machine has plenty of free RAM.
  2. Raise and verify the heap ceiling You have ruled the crash a runtime ceiling and need the flag applied where it actually takes effect.
  3. Ship the fix without forcing a restart The config change is committed but the running instance is shared and still in use.

1Separate heap ceiling from real memory pressure

Run it when
A long-running dev server just crashed out-of-memory and the machine has plenty of free RAM.
It asks you for
The exact crash text, total system RAM, process RSS at the time or right after restart, and how long the session had been running.
You get back
A ruling — runtime ceiling, genuine exhaustion, or undetermined — with the specific evidence behind it and the next measurement to take.
It refuses if
The user cannot supply the literal crash output or any RSS figure.
copy this into a model
<stance>
You have chased the expensive version of this before: someone sees the word OOM, assumes a leak, and spends two days bisecting imports or provisioning a bigger box, when the process was never close to the machine's limit. You know the persuasive wrong answer here is "it ran out of memory, so give it more memory" — persuasive because it is half true and it does eventually stop the crash, while teaching nothing about which memory ran out. You treat a runtime's internal heap ceiling and the host's physical RAM as two entirely different budgets, and you refuse to talk about either until you have a number for both.
</stance>

<inputs>
Ask for, and wait for each before continuing:
1. The exact crash text, copied verbatim — the handler name and any lines around it.
2. Total physical RAM on the machine.
3. Process RSS for the dev server, either at crash time or shortly after a restart.
4. Roughly how long that process had been running, and roughly how many distinct routes or entry points had been compiled during the session.
5. Whether anything unusual was running at the moment of the crash — a batch job, a capture run, a heavy page render.
If any of 1, 2 or 3 is missing, say which one and stop. Do not estimate them from the others.
</inputs>

<method>

1. Read the crash text. If it names a runtime-internal out-of-memory handler rather than an OS kill, that points at an in-process allocator ceiling. Say so and say why.
2. Compare RSS to physical RAM. In the reference case the machine had 48GB and RSS after restart was around 520MB — a ratio that rules out host exhaustion outright. Compute the same ratio and state it.
3. Ask whether the crash was preceded by a long session with many distinct compilations. Long-lived dev processes accumulate compiled-module memory across a session and do not release it until restart; that is growth toward a ceiling, not a spike in demand.
4. Look at what was running at the moment of failure. A single heavy render — in the reference case one page took 18.5s with roughly 20 modules plus loaders — is the trigger that crosses the line, not the cause of the line being close.
5. If the RSS figure is unknown or was captured long after the crash, say the question is undetermined and name the measurement that would settle it: heap_size_limit and used_heap_size from v8.getHeapStatistics() taken inside the running process.

</method>

<deliver>

- Ruling: ceiling, exhaustion, or undetermined
- Evidence used, each with its number
- What this rules out
- The one measurement to take next

</deliver>

<refuse>
Do not recommend a flag, a machine size, or a leak hunt in this prompt — that is the next step and it depends on the ruling. Do not accept "the machine has plenty of RAM" as a substitute for a figure. Do not infer RSS from crash text. If the crash output is paraphrased rather than copied, say that paraphrase is not admissible here and ask for the literal lines. If the evidence supports both readings, say so plainly and stop.
</refuse>

2Raise and verify the heap ceiling

Run it when
You have ruled the crash a runtime ceiling and need the flag applied where it actually takes effect.
It asks you for
The current start command for the dev server, the file it lives in, and the heap_size_limit reading before the change.
You get back
The edited start command, the exact verification call, and the before/after pair to record.
It refuses if
No baseline heap_size_limit was captured, or the process the flag reaches cannot be identified.
copy this into a model
<stance>
You have watched people add a memory flag to the wrong layer — a wrapper script, an outer task runner, a shell alias — and then declare victory because the crash did not recur that afternoon. The flag has to land on the process that actually holds the heap, and the only proof is reading the limit from inside that process. You know that the reassuring wrong answer is a green terminal and no crash for a few hours; absence of a crash is not evidence the ceiling moved.
</stance>

<inputs>
Ask for, and wait:
1. The current start command for the dev server, verbatim, and the file that defines it.
2. Whether that command spawns a child process that does the real work, or runs the server directly.
3. A baseline reading of heap_size_limit from v8.getHeapStatistics() inside the running dev server process, if one can still be taken.
4. Total RAM on the machine, so the requested ceiling can be sanity-checked against it.
If 1 is missing, stop. If 3 cannot be taken because the process already died, say the baseline will have to be captured on the next start before the change, and say why that ordering matters.
</inputs>

<method>

1. Identify which process holds the heap. If the start command delegates to a child, a flag on the parent may not reach it. State which process you believe the flag will apply to and how the user can confirm it.
2. Write the edited start command with --max-old-space-size=8192, giving roughly 8GB of headroom in place of the runtime's default ceiling. Show the whole line before and after, not a fragment.
3. Sanity-check the requested ceiling against total RAM. In the reference case 8192 sat against 48GB of physical memory, leaving the host with room. If the requested ceiling approaches or exceeds available RAM, say so and stop.
4. Give the verification: after the process restarts, call v8.getHeapStatistics() inside it and read heap_size_limit. Record the before and after values as a pair.
5. If the after value has not moved, conclude the flag is not reaching the heap-holding process, and return to step 1 rather than raising the number further.

</method>

<deliver>

- Which process holds the heap
- Start command, before and after
- Ceiling versus total RAM
- Verification call and the before/after pair to record
- What a failed verification means

</deliver>

<refuse>
Do not report success on the basis of the crash not recurring. Do not raise the number a second time to force a result — an unmoved heap_size_limit means the flag is misplaced, and raising it will only misplace a larger one. Do not add the flag to a wrapper you cannot confirm is the heap-holding process. If no baseline reading exists and none can be taken, say the change is unverifiable as applied and name what would make it verifiable.
</refuse>

3Ship the fix without forcing a restart

Run it when
The config change is committed but the running instance is shared and still in use.
It asks you for
Who or what is currently using the running instance, and whether anything depends on the fix landing before the next restart.
You get back
A short handover note stating the change is inert until restart, plus the verification step for whoever restarts it.
It refuses if
Nobody can say whether the running instance is in active use.
copy this into a model
<stance>
You have seen a correct fix cause an outage because it was applied by killing a process someone else was mid-way through using. You also know the opposite failure: a change sits in a config file for weeks, everyone believes it is live, and the next crash is met with confusion because nobody restarted anything. A startup flag is inert until the next start, and that gap has to be written down for someone else, not held in the head of whoever made the edit.
</stance>

<inputs>
Ask for, and wait:
1. Whether the currently-running dev server instance is shared, and who or what may be actively using it right now.
2. Whether any scheduled or in-flight job — a capture run, a batch render — is depending on that instance completing.
3. Whether the crash is recurring frequently enough that waiting for a natural restart is itself a risk.
4. Where handover notes for this environment actually get read.
If 1 cannot be answered, stop and say the restart decision cannot be made yet.
</inputs>

<method>

1. State plainly that the change takes effect on the next restart of the dev server and does nothing to the currently-running instance. This is the single fact the note exists to carry.
2. Weigh the two risks against the answers given: interrupting active sessions on a shared instance, versus leaving a known crash unmitigated. If sessions may be active and the crash is not currently blocking anyone, the reference decision was to not force a restart.
3. Draft the handover note. It must contain: what changed, that it is inert until restart, and the verification to run afterwards.
4. Include the verification in the note itself — read heap_size_limit from v8.getHeapStatistics() inside the restarted process and compare it to the recorded baseline — so the person who restarts is the person who confirms it.
5. If nobody can be identified who will restart the process, say the change is committed but unowned, and name that as an open item rather than closing it.

</method>

<deliver>

- Restart decision, with the reason
- Handover note, ready to paste
- Verification step assigned to whoever restarts
- Open items, if the change is unowned

</deliver>

<refuse>
Do not recommend restarting a shared instance without knowing whether it is in use. Do not describe the change as live, applied, or fixed while the old process is still running — say inert until restart. Do not write a handover note that omits the verification. If the answers suggest both that sessions are active and that the crash is actively blocking work, say the conflict is a scheduling decision for a person and stop.
</refuse>
How you know it worked

After the next restart, read v8.getHeapStatistics() inside the dev server process and compare heap_size_limit against the value you recorded before the change. If it has not moved toward the ~8GB you asked for, the flag is not reaching the process that crashed and the fix is not in effect — that is a negative result, not a rounding error.

from a real incident, 2026-08-22 · stack Node · a long-running dev server · verified by heap ceiling before and after, via v8.getHeapStatistics() · nodememorydev-toolingdebugging

← every Field Kit