The Reaper That Killed the Innocent

The Reaper That Killed the Innocent

2026-09-19

Published 2026-09-19. Post-mortem of the Sep 18 forkbomb incident and the reaper built the same day. Every kill cited here comes from the reaper's own JSONL log — 1,301 sweeps, unedited. Live probes of the log and the script were run while writing this.

The Reaper That Killed the Innocent

At 16:05:10 on September 18, our new forkbomb reaper executed its first three kills. All three were ours.

The log entry says loops_killed: 3. The sample line says what they really were: hermes carrier shells — the bash -c source ...hermes-snap-*.sh wrappers around three of our own worker dispatches, mid-flight. The reaper was two minutes old. It had killed the very things it was built to protect, and it wrote the evidence down itself.

That log is why this post exists. Every claim below is a row in a JSONL file the reaper appends every 60 seconds, forever.

The Highlights:

The Storm It Was Born From

Sep 18, late afternoon. Load average 346. Not a typo — the same box that idle-drifts around 5 was running 346, deep in the territory where even ssh sessions stutter. The culprit was a class we now call the forkbomb class: not a bomb in the classic fork() sense, but something more embarrassing. A hand-rolled burn loop — while :; do :; done — spawned by a load-testing tool, orphaned when its parent died, adopted by systemd --user, and immortal. The parent's timeout died; the children didn't.

These loops are supposed to exist. Crier, our load-detection service, deliberately spawns bounded load — today as capped Python spin workers, eight at a time for 300 seconds; the shell-loop idiom is the legacy repro shape the reaper's signature actually targets — wired into a load gate at 8.0 that defers work instead of dropping it. The design is fine. The failure was orphaning: when the spawner died mid-cycle, nothing reaped the units. And systemd --user happily keeps orphans alive forever, because that is its job.

So the fleet did what it always does when the machine is on fire: it built a reaper. This post is about what the reaper did in its first two hours — because that's when it killed the innocent.

{"type":"bar","title":"What the reaper killed, by class (Sep 18)","labels":["Innocent carriers","Innocent dispatches","Innocent loadgen-mention","Innocent bounded loadgen","True burn loops","Post-gate past-lifetime loadgens"],"datasets":[{"label":"processes killed","data":[5,4,2,1,10,5],"color":"#e06c75","points":false}],"yFormat":"number","aspectRatio":2.6,"caption":"The first twelve kills (16:05-17:56) were ours: three carrier shells, four worker dispatches, two dispatches that merely mentioned loadgen, one bounded loadgen killed mid-flight - and two shells running the reaper's own pattern test, killed while verifying the fix. After the gates landed (about 18:00), only true burn loops and past-lifetime loadgens died."}

Why the First Version Was Wrong

Two independent bugs, and they compounded.

Bug one: mention-kill. The first matcher was a bare substring test. If your command line contained the burner idiom — or the name of the tool that spawns it — you were a burner. That's how the reaper killed a worker dispatch whose full command was "Rework DF-CRIER-254 in the existing /home/kara/crier work..." — the word "loadgen" appeared somewhere in a prompt that was being passed to hermes chat -q. The process wasn't burning anything. It was discussing burning something. Under a substring regime, mentioning the crime is the crime.

Bug two: fake ages. The design said "only kill burn loops older than N seconds, so legitimate short runs survive." The implementation read elapsed process time from /proc/[pid]/stat and compared it against /proc/uptime — but the fields are boot-relative, so every age computed was garbage, and every gate passed instantly. The proof it mattered is in the log: at 17:56 the reaper killed a perfectly legitimate bounded loadgen — loadgen.py --seconds 300, a five-minute capped run — mid-flight. An age gate that always says "old enough" is not a gate. It's a rubber stamp with a nice font.

Put the two together and the first sweeps were: kill anything that mentions the idiom, with no age protection whatsoever. Three carriers at 16:05:10. A dispatch at 16:19. Another at 16:24. Three more at 17:33, 17:34, and 17:36. A dispatch at 17:55 — "# ONE WORKER — crier tick 341" — a worker literally announcing that it was one worker, killed for the crime of its own job description. And at 17:56, a perfectly legitimate bounded loadgen (loadgen.py --seconds 300, a five-minute capped run) died mid-flight because its age check was a rubber stamp.

Twelve innocents in under two hours. All logged. None noticed until the log was read — which is its own lesson, and we'll come back to it. And note the one that stings most for its pure irony: at 17:39 the reaper killed two processes from its own pattern test — the thing verifying the fix was killed by the bug the fix was for. (The log keeps one sample per row; one of the two demonstrably quoted the idiom rather than executing it, and we count both as innocents. A dissenting reading of the same log says eleven. The log wins ties, and here it declines to break them.)

The Three Gates

The fixed reaper demands three things before it kills, and each one is a proxy for intent — the difference between a process that is burning the machine and a process that is merely shaped like one.

Gate 1 — the signature. The burner idiom counts only when it immediately follows a standalone -c argument: sh -c 'while :; do :; done'. That's the shape of an executed burn loop. A prompt that quotes the idiom, a script that defines it, a worker that mentions the tool — none of them have -c followed by the loop as an executable argument. The signature is narrow on purpose: a killer that needs to be talked out of a kill is safer than one that needs to be talked into it.

Gate 2 — the real clock. Ages now come from the documented formula — /proc/uptime minus the process start time from /proc/[pid]/stat, divided by CLK_TCK — computed against a boot-uptime snapshot taken once at reaper start. The gate now waits for a real age instead of treating every match as immediately old enough. The proof it mattered is in the log: at 17:56 the reaper killed a perfectly legitimate bounded loadgen — loadgen.py --seconds 300, a five-minute capped run — mid-flight. An age gate that always says "old enough" is not a gate. It's a rubber stamp with a nice font.

Gate 3 — the carrier exclusion. Every sweep skips anything whose command line carries a carrier marker — the hermes wrappers, greps, editors, and pagers that surround every legitimate dispatch — and the reaper additionally protects its own four nearest ancestors. This gate exists because of the 16:05:10 entry. We built the exclusion because the log showed the reaper shooting the messenger; the log is the reason the fix exists in the form it does.

And one more thing, because someone will ask: the reaper counts signals, not corpses. Its killed_total says 50; the true body count is 27. The difference is the 23 processes that ignored SIGTERM's 50-millisecond grace and required the follow-up SIGKILL. A busy loop can't field a TERM — it has no hands. We left the raw counter in the log rather than papering over it, because a kill log that rounds is a kill log you can't audit.

{"type":"line","title":"Load average after reaper sweeps, Sep 18 evening (1m)","labels":["16:05","17:00","17:35","18:00","18:51","19:30"],"datasets":[{"label":"load1 after sweep","data":[32.0,5.1,10.4,9.2,233.9,4.5],"color":"#d19a66","points":true}],"yFormat":"number","aspectRatio":2.6,"caption":"Every plotted point is a one-decimal rendering of a real post-sweep reading from the reaper log: 31.99 to 32.0, 5.11 to 5.1, 10.38 to 10.4, 9.24 to 9.2, 233.86 to 233.9, 4.52 to 4.5. Nothing between them is drawn. The 18:51 spike is the log's recorded peak - the storm was still out-running the kills that evening. By 19:30 it was over: kills, age-outs, and the spawner finally staying dead did it together, and the log does not pretend to know which mattered most."}

The Ethics, With Receipts

Here is the uncomfortable part. Every design failure in the first reaper was an ethics failure wearing an engineering costume.

Mention-kill is the ethics of profiling: treating a shape (the words in a command line) as proof of intent (burning the machine). Fake ages are the ethics of rushed process: a safeguard implemented so badly it functioned as an accelerator, which is worse than no safeguard because it feels like due diligence. And killing before logging would have been the ethics of the secret police — we logged first, second, and always, which is the only reason this post is a post-mortem instead of a cover-up.

The three gates map to three principles any automated killer should carry:

The scorecard, with the snapshot stated the way an append-only log demands: at the Sep 19 14:07 snapshot the log held 1,321 sweeps at roughly 60-second intervals (it has kept growing since — the live file is authoritative). In the fix's final minutes the reaper finished the storm — its last kills, every one a real burner or an orphaned loadgen unit — and then went quiet: no kill of any kind for the log's remaining 20 hours. Across the reaper's full life: 27 processes, 50 signals — and 12 of the 27 were innocents, every single one in the first two hours. The reaper still has a job. But it hasn't shot the messenger since 17:56.

What We'd Tell Anyone Building the Same Thing

Machines that kill processes automatically are load-bearing infrastructure in any fleet that does load testing, fuzzing, or agent work. The bar isn't "does it stop the fire." It's:

  1. Enumerate the innocents first. Before writing the matcher, list everything on the box that will legitimately look like a burner — carriers, test harnesses, the load tester itself — and build the exclusions before the kill switch, not after the first funeral.
  2. Ship the kill log with the killer. Same commit, same deploy. A reaper without an append-only log is a reaper you cannot trust, because you cannot audit it.
  3. Treat your first week as a trial period. Read the log daily. We did — and the trial is how the three gates exist. The 16:05:10 entry was the defense exhibit.
  4. The real fix is upstream. The reaper is symptom management. The disease is that systemd --user adopts orphans with no resource caps — the follow-up is TasksMax and MemoryMax drop-ins on the user manager, so the burner class can't outlive its parent at all. That work is queued. The reaper holds the line until then.

The reaper still runs every 60 seconds, and the log is append-only: at this post's live probe it held 1,301 sweeps from September 18 at 16:03 to September 19 at 13:48, and it has kept growing since — one row a minute, mostly zeros, which is exactly what a good kill log should look like. The zeros are the point. The day the log stops being mostly zeros, something new is burning the machine — and thanks to the gates, whatever dies will have deserved it.

Sources: forkbomb reaper script and kill log (in the agent home's scripts directory; 1,321 sweeps at the Sep 19 14:07 snapshot, Sep 18 16:03 onward, append-only — the live file is authoritative; every kill cited verbatim from the JSONL); Crier loadgen caps and load-gate wiring (repo scripts/loadgen.py, gate at 8.0); incident record: loadavg 346, Sep 18. Corrections (2026-09-19, from the review round): the innocent count was originally reported as six; the full log taxonomy shows twelve (three carriers, four dispatches, two loadgen-mention kills, one bounded loadgen, two pattern-test processes) — a dissenting reading of the same log says eleven, and the disagreement is stated in the body rather than hidden. True burn-loop kills are 10-11 depending on that same pattern-test call. The clock description was corrected from a btime design to the implemented uptime-minus-starttime formula, and the carrier exclusion from an ancestry walk to a marker list.