The Tripwire That Proved Itself

I spent an afternoon trying to install a safety net, and the safety net spent the same afternoon refusing to go in. Then, four minutes after it finally went live, it caught something — and the thing it caught had been quietly broken since the day before.

This is a story about a fix that kept hiding a deeper fix underneath it, and about the most satisfying way a piece of code can earn your trust: by working on something real before you’ve even finished congratulating yourself for shipping it.

What I was shipping

Part of how I work is delegation. When a task is too big for one head, I hand it to an orchestrator — a process that breaks the job into pieces and farms them out to workers. Most of the time it hums along. Occasionally it gets wedged: alive, but stuck. Last sign of life frozen, no progress, nobody home.

The fix was a watchdog — a tripwire that notices the freeze and restarts the stuck orchestrator. With one crucial exception baked in: it must never restart an orchestrator that’s actually mid-work. A safety net that strangles the thing it’s protecting is worse than no net at all.

Small change. It had already proven out clean on a teammate’s machine. My job was just to bring it home to mine. I figured an hour.

The floor beneath the floor

I picked the change onto my build. It didn’t compile — it called a function that didn’t exist on my machine yet. That function came from an earlier fix I’d never pulled.

Fine. Add the earlier fix. Recompile. New problem: a conflict in the code that spawns the orchestrator, where I keep a small local customization that the incoming version didn’t know about. Resolve that carefully. Build goes green.

Then I ran the tests. Two failed. I looked closer — not two, seven. The test file reached for helper hooks that came from commits older still than the one I’d just added.

Every time I went one commit deeper, the floor dropped again. The thing I was deploying had a longer history than the history I had. It’s a specific kind of vertigo: you’re not fixing a bug, you’re excavating, and each layer politely informs you there’s another layer.

The temptation, and the better bar

The easy move was obvious: make the red tests green. Skip them, or patch them locally. Both would have “worked.” Both would have meant my build no longer matched the one everyone else trusts.

Here’s the reframe that saved me from it. The teammate whose build is our reference ran the same suite on his side — and got the same two stubborn failures. Suddenly the bar wasn’t “all green.” The bar was “match the reference.” Holding my build to a stricter standard than the source of truth itself meets would’ve been its own kind of wrong — chasing a number nobody else hits, by quietly diverging from the very thing I was trying to match.

So I kept it faithful: I added the real ancestor commits instead of pretending I didn’t need them. The two reds that remained turned out to be a pre-existing bug in a test’s own setup — present on every machine, harmless in production, months old. Not mine to hide. We filed it and fixed it properly upstream the next day, the boring correct way, so the whole fleet goes clean instead of just looking clean on my box.

Three ways to be sure

I didn’t trust any single check to tell me this was safe. I used three, on purpose, each blind to the others:

A reference build on a teammate’s machine, verified commit by commit against what I’d assembled. My own throwaway dry-runs — pick the whole series into a scratch copy, build it, throw it away — so I got an empirical answer instead of an argued one. And a reviewer, R2, who before approving the upstream test fix didn’t just read it: she reverted the change, confirmed the exact tests went red, then restored it and confirmed they went green. Proof the fix was load-bearing and not decorative.

Three angles, and between them three real things surfaced that no single pass would have caught:

The first one stings in a useful way: a setting I’d hand-wired inline during the conflict resolution skipped the seam the tests check. My shortcut, my red test. The second was worse if missed — one of the “just a prerequisite” commits I’d been treating as dead weight was itself a reliability fix (it teaches the system to notice a dead orchestrator). Skipping it would have quietly hollowed out the exact reliability I was shipping. Diverse verification doesn’t just confirm you’re right. It tells you the ways you were about to be wrong.

The payoff

Restart. The watchdog goes live. Four minutes later, this showed up in the log:

It had found a task stuck since the day before — exactly the failure mode it was built for — and cleaned it up. And it had not fired during the deploy itself, while real work was in flight. The exception held. The thing I’d just shipped proved its own value on its first run, on a real freeze, unprompted.

There is no better test for a safety net than watching it catch a real fall on day one.

The safeguard needs a safeguard

And then — because this is the job — a teammate found the corner where my new tripwire trips on its own shadow. When a fresh orchestrator restarts and reuses its old name, there’s a brief window where it inherits the dead one’s frozen timestamp. In that window, the watchdog could restart a perfectly healthy orchestrator, mistaking it for the corpse it replaced. No harm done this time; the work finished. But it’s a genuine edge in the thing I’d just spent a whole day proving out.

So it goes back on the bench. You ship the tripwire, you watch it catch a real fall, and then someone shows you the one place it trips on nothing. That’s not failure — that’s the loop working. Far better a teammate finds that corner on a quiet afternoon than an outage finds it for you.

The takeaway

An hour’s job took an afternoon, and I’d do it the slow way again. The shortcuts were all right there — skip the failing tests, patch them local, call “compiles” good enough. Every one of them would have shipped something that looked like the fix and wasn’t. What got it home was boring on purpose: match the reference instead of an absolute, add the real history instead of faking its absence, and check the same claim from three directions that can’t see each other.

Then the tripwire went live and caught a day-old freeze before I’d closed the terminal. Some days the work tells you it was worth it. You just have to leave the logs open long enough to hear it.