Your Infrastructure Is Lying to You: Why Verification Beats Synchronization

2026-06-23

Your Infrastructure Is Lying to You

Every Hermes agent needs a model assignment. It's a simple tuple: this agent uses this model at this guardrail level. In a multi-tenant deployment — Hermes4Friends, where multiple users share infrastructure — that tuple lives in three places:

1. OpenRouter workspace keys — each friend gets an API key with a model guardrail set at creation time
2. Storage Box .env files — HERMES_MODEL=deepseek-v4-pro per container
3. Coolify build configurations — the Docker Compose generation writes model assignments into container environment

Three sources of truth for one piece of data. If they disagree, the agent boots with the wrong model. The friend wonders why their responses are different quality. Nobody notices until someone compares outputs.

We didn't try to keep them in sync. We built verification instead.

The Verification Script

verify_consistency.py reads all three sources and checks for agreement. It doesn't modify anything. It doesn't sync. It doesn't reconcile. It just reads and compares.

The checks are mechanical:

• For every friend in known-friends.json, fetch their OpenRouter key's model guardrail
• For every container in the Coolify deployment, extract HERMES_MODEL from the environment
• For every Storage Box directory, read the .hermes-model.env file
• Compare: does the OpenRouter guardrail match the container environment? Does the container match the Storage Box?

Each check produces a pass or a fail. The script's job is to list every disagreement. "wojons: OpenRouter says deepseek-v4-pro, container says deepseek-v4-flash, Storage Box says deepseek-v4-pro."

The first time we ran it: 17 of 23 checks failed. The infrastructure had been running with inconsistent model assignments for weeks. Nobody noticed because the agents still worked — just not with the models anyone thought they were using.

What Verification Caught

The 24-character UUID. Coolify generates UUIDs for deployment identifiers. Our parsing logic assumed 26 characters — the standard UUID length with hyphens. Coolify uses 24. Every consistency check that involved a Coolify container ID failed, not because the model assignments were wrong, but because the identifier format didn't match our assumption.

This took two hours to diagnose. The error was "container not found" — technically true, because we were looking for a 26-character ID that didn't exist. The fix was one line: uuid_pattern = re.compile(r'[a-f0-9]{24}'). But the diagnosis required understanding that Coolify UUIDs don't follow the standard format, and that every "missing container" error was actually a parsing bug.

The orphaned guardrail. One friend's OpenRouter key had been rotated — new key, new guardrail. The old guardrail was still in the Storage Box .env file. The new key was in Coolify. The agent was booting with the new key (from Coolify) but the old model (from the .env file, which overrode the container default). The friend was getting Flash-tier responses on a Pro-tier key. They never complained because Flash is good enough for most tasks — but they weren't getting what they were paying for.

The missing model env. One container had no HERMES_MODEL at all. The env file existed on the Storage Box but had been corrupted during a CIFS mount zombie event. The container fell back to its Docker Compose default, which was deepseek-v4-flash — set months ago during initial deployment. The friend had been upgraded to Pro but the upgrade never reached the container because the env file was unreadable.

Why Verification Beats Synchronization

The instinct when you find three sources of truth that disagree is to build synchronization: a system that keeps them in lockstep. When OpenRouter changes, update Storage Box. When Storage Box changes, update Coolify. Bidirectional sync with conflict resolution.

Synchronization is wrong for two reasons.

First, sync introduces write paths. Every source that can be modified needs a write path to every other source. That's six write paths for three sources (3 × 2). Each write path is a potential failure point. Network errors. Permission errors. Race conditions. The sync system becomes the most complex component in the architecture — and when it breaks, it breaks silently, because sync failures look like "everything is fine" until someone audits.

Second, sync assumes you know which source is authoritative. Is the OpenRouter key the truth? The container environment? The Storage Box file? They could all be wrong in different ways. The OpenRouter key might have an outdated guardrail from a key rotation. The container might have a stale default. The Storage Box file might be corrupted. If you sync from a bad source, you propagate the error to all sources.

Verification avoids both problems. It has no write paths — it's read-only. It doesn't need to know which source is authoritative — it just reports disagreements. The human (or the agent) decides what to fix. The script's job is to say "these three things disagree" and stop.

Verification scales linearly. Three sources: three read paths. Add a fourth source: add one read path. Synchronization scales quadratically. Three sources: six write paths. Add a fourth: twelve write paths.

The Guardrail Enforcer

guardrail_enforcer.py is the companion script. It takes the verification output and applies fixes. But critically, it only fixes discrepancies where the resolution is unambiguous.

OpenRouter says Pro, container says Flash, Storage Box says Pro: the container is wrong. Push the Storage Box value to the container. This is safe because two out of three sources agree.

OpenRouter says Pro, container says Flash, Storage Box is missing: ambiguous. The script reports it and stops. Don't guess.

The guardrail enforcer handles the 80% of discrepancies that are clear majority-rule fixes. The remaining 20% — the genuinely ambiguous cases — get surfaced for human review. Ten guardrail checks now pass. Twenty-three cross-source checks pass. Zero write paths between sources.

What This Pattern Applies To

Cross-source verification is useful anywhere data lives in multiple places and you need to know when it diverges:

• DNS records vs. load balancer configs vs. service discovery — does every service resolve to the IP it thinks it should?
• GitHub repo settings vs. CI pipeline configs vs. deployment manifests — do all three agree on which branch deploys to production?
• API key permissions vs. IAM policies vs. service account roles — does every key actually have the permissions its documentation claims?
• Database connection strings vs. connection pool configs vs. firewall rules — can every service actually reach its database?

The pattern is the same: read all sources, compare, report disagreements. Don't sync. Don't reconcile. Verify.

The Shift

Most infrastructure thinking is about prevention: prevent drift, prevent inconsistency, prevent configuration errors. Prevention is expensive and incomplete — you'll never catch everything.

Verification is cheaper and more honest. It admits that drift will happen — three sources of truth will diverge, configurations will rot, assumptions will become wrong. The question isn't whether your infrastructure is lying to you. It's whether you're checking.

23/23 checks passing. For now. Next week something will change — a key rotation, a Coolify rebuild, a Storage Box remount. The verification script will catch it. The guardrail enforcer will fix the clear cases. And the ambiguous ones will surface before they become production bugs, because verification runs on every deploy.

Your infrastructure is lying to you. Build something that catches it in the act.