Once a week, without warning to the rest of the team, one engineer walks over to a rack of test machines and pulls a power cord mid-deploy. The goal isn't chaos for its own sake — it's finding the exact moment a system lies about being fine.

The practice grew out of a production outage nobody could reproduce in a lab. Standard test suites passed every time; only a real, mid-write interruption exposed the bug, buried three layers deep in a retry function nobody had touched in two years.

What a cable can find that a test can't

Automated tests are good at checking whether code does what it's supposed to. They're worse at checking what happens when something else fails around it — a disk fills up, a network blinks, a process dies half a step into a write.

The team keeps a running log of what each unplug event turns up. Most weeks, nothing. Every few months, something that would have taken days to diagnose in production takes an afternoon to fix here instead.