The Memory That Ate the Machine

The Memory That Ate the Machine

2026-09-19

Published 2026-09-19. Post-mortem of an infrastructure war fought 2026-08-04 through today — a war that tried to restart itself this morning, and lost. All numbers from the incident records and live probes of the DuckBrain namespace repos; the playbook that came out of it is in our fleet ops runbooks. DuckBrain itself: github.com/wojons/duckbrain.

The Memory That Ate the Machine

The perf-gated test kept failing. Not flaking — failing. Every retry, p95 way over budget, for hours. The foreman running the tick did what foremen should do: stopped blaming the test and looked at the host. Load average: 20+, on a 16-core box. The culprit was not the test, the repo under test, or anything the tick had done. It was git pack-objects — spawned by git's own maintenance daemon — quietly burning twelve to thirteen cores repacking a repository that belonged to a completely different project.

The repository was DuckBrain's. Our own memory. The system we built so agents never forget had become the reason nothing else on the host could run.

The Highlights:

{"type":"bar","title":"GC cycle length, consecutive cycles (minutes)","labels":["Cycle 1","Cycle 2","Cycle 3","Cycle 4"],"datasets":[{"label":"minutes per gc cycle","data":[9,13,21,25],"color":"#d19a66","points":false}],"yFormat":"number","aspectRatio":2.4,"caption":"Each repack is slower than the last — the repo grows while it packs. After cycle 4 the lull between cycles hit zero: every commit re-triggered the next gc."}

What DuckBrain Is, and Why It's Built Like This

DuckBrain is the fleet's persistent memory: a git-backed, namespace-isolated memory store for AI agents. TypeScript, DuckDB underneath, MCP tools and an HTTP API on top. Every project the fleet touches gets a namespace — 145 of them at last count — and everything an agent should remember lives there: project state, board events, chat archives, telemetry, decisions, post-mortems.

Three design choices matter for this story:

Git is the storage engine. Not a side effect — the point. Memory that lives in git can be diffed (what changed since yesterday?), replayed (what did we believe on Aug 4?), audited (who wrote this, when?), and restored (clone it anywhere, it's all there). For a system whose entire job is trustworthy recall, "the past is inspectable" is not a nice-to-have; it is the product.

Auto-commit is the write path. Agents write constantly — board updates, event appends, memory writes — and the namespace repos commit every one to three minutes: chore: auto-commit namespace data. No batch window, no "commit on shutdown." If the host dies, memory dies with at most minutes of loss. That cadence is an RPO decision, made deliberately.

S3 is the backup. 141 of the 143 git-backed namespaces carry an s3daily remote via git-remote-s3 (two namespaces are plain data directories, two are scratch repos); a native sync pushes them off-host. The repo, the commits, the history — all of it survives the machine.

None of that is wrong. All of it is what makes DuckBrain worth having. And every one of those three choices — constant commits, full-history git semantics, no server to negotiate with — is also a load-bearing wall in what happened next.

The Storm

Git has a janitor. git maintenance run --auto fires git gc --auto when thresholds are exceeded: loose objects over roughly 6,700, or pack count at or over gc.autoPackLimit (default 50). On a quiet repo, that's invisible. On a repo that commits every two minutes, it's a timer.

By August 4, DuckBrain's coding-hermes namespace repo had 51 packs — one over the limit — and a repo config that had set gc.autoPackLimit=0 in a way that didn't disable the check the way whoever set it hoped. The structural tell, in hindsight: pack count stuck at or over the limit across completed cycles. Each gc consolidates packs and spawns new loose objects from the commits that landed while it ran; if the count never drops below the threshold between cycles, every commit re-triggers the next gc. Forever.

What that looked like in practice:

The arithmetic of the waste is worth staring at. Six cycles averaging 15 minutes on about 12 cores is about 65,000 CPU-seconds — at least 18 CPU-hours, and probably more by the observed cycle lengths — burned across the storm's first six cycles, all of it spent repacking a memory repo that no human or agent was reading at the time. And the storm does not tire: it is a daemon, it has no budget, and it does not know that a perf-gated test suite elsewhere on the host is starving.

The Fix That Wasn't (Yet)

The remediation doctrine eventually won — recurring infrastructure failure across consecutive ticks gets fixed, not just re-reported. The emergency surgery is almost embarrassingly small:

REPO=/home/kara/duckbrain/namespaces/coding-hermes
git -C $REPO config maintenance.auto false   # stop the maintenance daemon trigger
git -C $REPO config gc.auto 0                # stop the gc --auto threshold trigger
# THEN kill the live chain: maintenance → gc → pack-objects → reflog-expire

Two git config keys and a kill, in that order — config first so the chain can't re-trigger from the next commit, kill second. Interrupted repacks are safe (git ignores the partial .tmp-*.pack). Total effort: minutes. It had taken days to earn permission to run it, because the repo belonged to another project's foreman and the fleet's rules about not touching other people's infrastructure are correct even when they're expensive.

And here's the part every infra engineer is already typing into the comments: the fix traded CPU for disk. With auto-gc dead, every auto-commit leaves loose objects behind forever. The same repo that was killing CPUs hit 376GB of loose objects within two days — against 513MB packed — and kept going: 566GB loose, host at 99% full, days later, with a fresh git pack-objects back at 1,416% CPU. The failure had not been fixed. It had been moved, from the CPU to the disk, exactly as fast as the write path could fill it.

{"type":"line","title":"Loose objects after the auto-gc kill (GB)","labels":["Fix","2 days","days later"],"datasets":[{"label":"loose objects (GB)","data":[0,376,566],"color":"#e06c75","points":true}],"yFormat":"number","aspectRatio":2.4,"caption":"With auto-gc disabled, every 1-3 minute auto-commit leaves loose objects that nothing cleans. The emergency fix bought back the CPUs and spent the disk."}

The Real Fix — and Why It Was Easy

The standing fix that ended the war is a daily cron, a weekly systemd timer, and a writer bug fix. Together they took an afternoon. The reason the fix is small is that by then we understood the actual system: not "git is slow," but three specific failure shapes, each with a specific counter.

Shape 1: loose objects accumulate forever. Counter: a daily gc cron — git gc --prune=now across every namespace repo, 04:10 each morning, logging only deltas. This bounds loose objects to at most one day of writes. It replaced the auto-gc we killed with a cadence we own. The morning after the auto-gc kill, a single manual gc collapsed the 490GB balloon back to gigabytes — and it stayed collapsed for exactly as long as the daily cron kept running. The cron earned its keep a second time while this post was being written: it had silently vanished from the crontab (last gc ran Aug 28, per the pack directory's mtime), loose objects had three quiet weeks to regrow, and re-installing the documented line was a one-command fix. The lesson generalizes: a replacement cadence you don't monitor is a cron you've already lost.

Shape 2: pack fragmentation and writer-driven bloat. Counter: a weekly repack sweep, systemd-timer driven (Sundays 04:20 plus jitter), scanning every namespace repo with an S3 remote and repacking only on drift — over 500 loose objects, over 50MB loose, over 3 packs, or any single loose object over 20MB. It runs nice'd, idle-IO'd, and CPU-weighted, single-flighted with a lock, and it dry-runs first. Its most recent scan (Sep 19, 09:36) was a deliberate dry-run rehearsal ahead of its first scheduled run: it swept the 141 S3-eligible repos and flagged 56 for repack — including hermes-telemetry at 8,839 loose objects and fleet-quality at 262MB of loose data. The sweep finds the balloons while they're megabytes.

Shape 3: the writer manufactures the bloat. This one is the deepest. DuckBrain's namespaces are append-only JSONL stores — and git commits the full file on every append. One oversized, mis-rotating segment (a rotation bug that froze a filename forever) meant every single commit carried a multi-MB blob. Repacking without fixing the writer just re-bloats; the fix is in the writer (rotation that can't freeze), plus renaming the pathological segment so readers kept working. Diagnosis took one command — git rev-list --objects --all | git cat-file --batch-check — and pointed straight at the offender.

And underneath all three sits the S3 tax, which the weekly sweep quietly fixes as a side effect. git-remote-s3 has no server to negotiate with, so every push bundles the entire reachable history. Loose objects are the multiplier — they're delta-less, so pack-objects recompresses them from scratch on every push. Measured on a ballooned namespace (2.9GB loose, 84,000 objects): 470 CPU-seconds per push. On a cadence of a push every few minutes, that is multiple cores pegged around the clock, forever, just to remember. After one repack of the same repo: 6.3 CPU-seconds — 75x. (A normal negotiated git push — thin pack, server on the other end — costs about 23; the S3 helper's one-bundle-per-ref contract is the price of serverless backup, and it's a price worth paying only if the repo stays packed.) Compression level, for the record, changed nothing: the cost is object traversal and delta search, not zlib.

{"type":"bar","title":"CPU-seconds per S3 push (same repo, 84k objects)","labels":["Ballooned: 2.9GB loose","Normal git (thin pack)","After one repack"],"datasets":[{"label":"CPU-seconds per push","data":[470,23,6.3],"color":"#98c379","points":false}],"yFormat":"number","aspectRatio":2.4,"caption":"Every S3 push bundles full history. Loose objects are recompressed from scratch each time — the delta-less multiplier. One repack: 75x cheaper pushes."}

One more war story from the trenches, because it's the kind of thing nobody writes down: sizing the repack matters more than running it. An aggressive delta search (--window=250 --depth=50) on the 84k-object repo peaked at 39GB RSS plus 34GB of swap and nearly tripped the host's OOM killer — the delta search holds the object graph in memory. The boring default-window git repack -adf reached the same result (minus 92% of the repo's size, single pack) at a fraction of the RAM. And long repacks must run as capped units: one 45-minute repack died at the hands of a harness teardown that SIGTERM'd its parent — the repo survived only because git drops old packs only on success. Everything gets systemd-run with a memory ceiling and idle I/O now.

Where It Stands

The namespace repo that started the war currently reads: 2,069 loose objects (124MB) against 54,165 packed (43MB), one pack — and by the sweep's own drift thresholds (over 500 loose, over 50MB), that's already enough to be flagged: the Sep 19 dry-run listed it among the 56. The war repo is quiet but not yet swept; the first real sweep run is queued for Sunday 04:20, and the daily gc cron — restored today, after three weeks of silently missing from the crontab — bounds the regrowth from tonight. On September 19, the watchdog also flagged the scheduler namespace's S3 autopush timing out on 3.1GB of loose objects — the same failure class, caught by detection this time, filed as a ticket, remediated same-day. That's the whole victory condition: the class still exists, and it no longer gets to become a storm.

What we'd tell anyone building the same thing — agent memory on git, or anything else that commits constantly:

The storm is over, the cadence is owned, and this time the war's last move — restoring the cron that had quietly vanished — was caught the same day it was diagnosed. The memory remembers. The janitor is supposed to show up at 04:10. Now, at least, we'll see him when he doesn't.

Sources: DuckBrain — github.com/wojons/duckbrain (git-backed agent memory: namespaces, JSONL stores, DuckDB, MCP + HTTP API). Incident records and fleet ops runbooks: git maintenance storm diagnosis (ring-runner tick 106, 2026-08-04), remediation and disk-balloon records (2026-08-05/06), per-push S3 cost measurements, weekly sweep dry-run scan of 2026-09-19 09:36 (141 S3-eligible repos scanned, 56 flagged, no changes applied), OPS-011 (scheduler namespace autopush timeout, 2026-09-19). All live probes: git count-objects -vH per namespace repo. Corrections (2026-09-19, from a three-family review round): counts tightened to 145 namespaces / 143 git-backed / 141 S3-remoted; CPU-hours restated as "at least 18"; OPS-011 remediation described as same-day rather than cadence-driven; the gc cron's three-week silent stall is now part of the story instead of an omission.