Passante A Passante publication

En Passante

In Passing
Field Kit

The glob walked the folders the filter discarded

Decide to skip a directory at its boundary, because filtering after collection has already paid the I/O.

What happened

On 2026-09-01, an audit covering 60 repositories and 7 checks ran for over two minutes because the script collected every path first and discarded dependency folders afterwards, and skipping those folders during the walk brought the same audit down to 16 seconds.

2026-09-01 a filesystem traversal script across 60 repositories measured: audit wall clock: over two minutes, then 16 seconds
3 prompts, in order
  1. Measure the walk before optimizing A multi-repository audit takes minutes and the search looks slow.
  2. Prune directories during the walk Counters show ignorable folders dominate the directories entered.
  3. Prove pruning kept every finding The pruned walk is faster and the results must be shown intact.

1Measure the walk before optimizing

Run it when
A multi-repository audit takes minutes and the search looks slow.
It asks you for
The traversal code, the root directories, and one timed full run.
You get back
Counts of directories entered and files examined, grouped by top-level directory name.
It refuses if
No traversal code is supplied, or no run has been timed.
copy this into a model
<stance>
The walk is the cost, not the match. A traversal that enumerates every path under a dependency tree pays for I/O that a later filter throws away. Measurement comes first. The reading that misleads here is that the checks are slow, when the real expense is reaching the files at all. Report what the counters show and refuse to estimate what they do not.
</stance>

<inputs>

1. The traversal code, or the exact glob pattern the script uses.
2. The root directories the walk starts from. Give the count.
3. The wall clock duration of one complete run.
4. The directory names the audit treats as ignorable.
Wait for each answer. Do not invent a pattern or a root path. If an answer is missing, ask once more and stop there.
</stance>

<method>

1. Read the traversal code. Decide whether it collects paths first and filters second. Quote the line that does the filtering.
2. Add a counter for directories entered. Add a second counter for files examined.
3. Print both counters when the script exits. Keep the audit logic unchanged.
4. Run the script once. Record both counters next to the wall clock duration.
5. Group the entered directories by their top-level name. Report the five names with the highest counts.
6. If ignorable directories dominate the entered count, the walk owns the time. State that conclusion.
7. If no group dominates, the time may sit in the checks. Say the evidence does not point at traversal. Stop and ask for per-check timing instead.
8. Do not change the traversal in this step. Diagnosis and repair stay separate.

</method>

<deliver>

- Collect-then-filter: the quoted line, or absence of one
- Counters: directories entered, files examined
- Top groups: the five heaviest top-level names with counts
- Verdict: walk-bound, check-bound, or undetermined
- Missing evidence: what would settle an undetermined verdict

</deliver>

<refuse>

Do not name a bottleneck without counter output. Do not guess how many files sit under a dependency folder. Do not report a percentage the counters do not support. If the user supplies only the wall clock, say the run is slow and the cause is unknown. If the traversal code is unavailable, stop and ask for it. If the counters contradict the user's suspicion, report the counters and say so plainly.

</refuse>

2Prune directories during the walk

Run it when
Counters show ignorable folders dominate the directories entered.
It asks you for
The traversal function, the ignore list, and the baseline counts.
You get back
A rewritten walk that skips ignored directories before descending into them.
It refuses if
The ignore list is unnamed, or baseline counts are missing.
copy this into a model
<stance>
Prune at the boundary. A skip decided at the directory level means the cost inside never happens, so the saved work is the whole subtree. Post-filtering is correct and slow, because every ignored path still gets enumerated before it is dropped. The fix that looks clever but changes nothing is a tighter glob pattern that still descends into dependency trees. Move where the decision happens, not the pattern string.
</stance>

<inputs>

1. The traversal function or glob call, pasted in full.
2. The exact list of directory names to skip.
3. The baseline counts from the diagnosis step. Include directories entered.
4. Any directories that must be audited even when their name matches the skip list.
Wait for all four. Do not assume a conventional ignore list. If item 2 is missing, ask for it and stop.
</inputs>

<method>

1. Identify the point where the walk decides to descend. Name the function or the loop.
2. Rewrite that point so matched directory names are removed before descent. Keep the match on the directory name only.
3. Keep the old post-filter in place for now. It should become a no-op.
4. Run the script. Compare directories entered against the baseline count.
5. If the entered count did not drop, the prune sits in the wrong place. Report that and show where the walk still descends.
6. Check the exceptions from input 4. Confirm each one still appears in the entered directories.
7. If an exception is now skipped, add an explicit allow check before the name match.
8. Record the new wall clock duration beside the baseline duration.

</method>

<deliver>

- Decision point: the function or loop that was changed
- Patch: the rewritten descent logic
- Counts: directories entered, before and after
- Timing: wall clock, before and after
- Exceptions: each protected directory and whether it was still visited

</deliver>

<refuse>

Do not widen the skip list beyond the names the user gave. Do not prune by path depth or by file size. Do not claim a speedup without the new counters and timing. If the entered count stays flat, say the prune is not taking effect. If a protected directory disappears from the walk, stop and report the regression. Do not delete the post-filter in this step.

</refuse>

3Prove pruning kept every finding

Run it when
The pruned walk is faster and the results must be shown intact.
It asks you for
Findings from both runs, in a sorted comparable form.
You get back
A diff of the two finding sets beside the counts and timings.
It refuses if
Only one run's findings exist, or the roots differ between runs.
copy this into a model
<stance>
A faster run that finds less is not an optimization. Pruning removes paths from consideration, so it can remove real findings along with noise. Equivalence is the claim under test. The result that reads as success and is not is a green run with a shorter report, accepted because it finished quickly. Compare the finding sets first. Seconds come second.
</stance>

<inputs>

1. The findings from the unpruned run, written to a file.
2. The findings from the pruned run, written to a file.
3. The root directories used by each run. State them separately.
4. The number of checks the audit performs.
Wait for both files. Do not reconstruct one run's output from memory. If either file is missing, ask for a rerun and stop.
</inputs>

<method>

1. Confirm both runs used the same roots. If the roots differ, the comparison proves nothing. Stop and say so.
2. Normalise both files. Sort the lines. Strip timestamps and absolute path prefixes.
3. Diff the two sorted files. Report the count of lines present only in the unpruned output.
4. Report the count of lines present only in the pruned output.
5. For each line lost by pruning, name the directory it came from.
6. If a lost line comes from outside the skip list, the prune is too broad. Report it as a defect.
7. If a lost line comes from inside the skip list, state that the audit scope changed deliberately.
8. Confirm both runs reported results for every check. Count the checks in each output.
9. Only after the diff is empty of unexplained losses, report the two durations and the entered counts.

</method>

<deliver>

- Roots: the roots used by each run, and whether they match
- Diff: lines unique to each side, with counts
- Losses: each lost finding and its source directory
- Scope change: skipped directories the user accepted
- Performance: directories entered and wall clock, both runs
- Verdict: equivalent, narrowed on purpose, or broken

</deliver>

<refuse>

Do not call the runs equivalent while unexplained losses remain. Do not compare unsorted output. Do not report a speedup before the diff is accounted for. If the two runs used different roots, refuse the comparison. If one run is missing a check, say the outputs are not comparable and stop. Never explain a missing finding as noise without naming its directory.

</refuse>
How you know it worked

Make the script print how many directories it enters. Run it with pruning and without. The pruned run enters fewer directories, finishes sooner, and returns the same findings.

from a real incident, 2026-09-01 · stack a filesystem traversal script across 60 repositories · verified by audit wall clock: over two minutes, then 16 seconds · filesystemperformanceautomationaudits

← every Field Kit