How a database connection limit almost rebooted my customers' Macs

We run a small fleet of Mac minis for customers. A daemon on a hub polls each machine over SSH, posts telemetry to a control plane, and escalates when a machine goes quiet. First it waits. Then it power cycles the machine through a smart plug. Then it asks for a human.

Last week the control plane started returning 500s. Thirteen minutes later the daemon had concluded that all four Macs were dead and logged a power cycle for each one. Two belonged to paying customers. A third had been sold that morning.

What actually failed

A database connection pool. The control plane runs on serverless functions against a Postgres pooler with a project-wide cap of fifteen connections. A few warm functions used them all, and every database-backed API route began failing. The details are in a separate post.

The Macs were fine. SSH worked. Every GUI session was up. The customer building on one of them noticed nothing. The only thing that was broken was the system whose job was to know whether the Macs were broken.

How a healthy fleet looked dead

The daemon kept one timestamp per machine: the last time a poll succeeded. The escalation logic compared that to now. The poll, reasonably enough, began by fetching the list of machines from the control plane, and returned early when that fetch failed.

So during the outage no poll ran, no timestamp advanced, and every gap grew together. The escalation logic did not know that the daemon had never tried. It saw thirteen minutes of silence from four machines and did what it was built to do.

ladder: Unit 02 unreachable 12m51s → power cycle
ladder: Unit 03 unreachable 12m51s → power cycle
ladder: Unit 04 unreachable 12m51s → power cycle
ladder: Unit 05 unreachable 12m43s → power cycle

Why nothing rebooted

No smart plug was configured. The power cycle step checked for a plug, found none, and moved on. Afterwards the four machines showed uptimes of six days, six days, one day and about an hour. Nothing had been touched.

That is luck, not design. Smart plugs are part of the plan for the next batch of machines. Had they been fitted, a database connection problem would have cut mains power to three customer Macs. One of them was possibly mid-build. None of them had done anything wrong.

The fix

The daemon now records when it last managed to fetch the unit list and actually try the machines. The escalation logic refuses to act unless that happened recently, and it logs why: the fault is the control plane, not the units.

ladder: skipped, no successful unit poll in 14m02s (control plane, not the units)

That line says exactly what it is refusing to do and why. It is what the old one should have said the first time.

The rule underneath is simple. A remediation must require evidence that the check ran. "We tried to reach it and failed" is evidence. "We have not heard from it" is not, because it is also what an outage of the listener looks like.

What to test in your own automation

We had tested what happens when a Mac dies. We had never tested what happens when the thing watching the Macs dies. Those are not the same test, and the second one is the one that reaches for the power switch.