Checks that cannot fail: five false positives from a week of Mac automation

One rule would have saved us most of a week. Before you trust a check, ask what it prints when the thing is broken. If the answer is "the same thing", you do not have a check. You have a decoration that says OK.

Here are five we found in one week, all on macOS, all in automation that had run without complaint for months. Each one produced a confident wrong answer, and each sent a human off to fix something that was not broken.

1. The update that was not running

A machine seemed slow to come back after a reboot. A quick pgrep -f softwareupdate found a match, so the script reported that a software update was in progress and we waited.

There was no update. softwareupdated is a resident daemon on every Mac, and a substring match finds it whether or not anything is happening. The owner of the machine caught it with a plain question: there should not be any update happening, so what is the issue? The check had no answer, because it never had one.

The fix is to look for what a running update leaves behind. A download in progress or sustained disk activity is evidence. A process name that is always present is not.

2. The Xcode that copied in zero seconds

We clone Xcode between machines by streaming a tar archive over SSH. One run piped it through a laptop that ran out of memory. The sending side died, the receiving tar -xf - read an empty stream, extracted nothing, and exited zero. The script printed XCODE_COPIED.

Zero bytes had moved. The exit code was honest about what tar had been asked to do, which was extract whatever arrived. Nothing arrived.

# what we checked
tar -xf - -C /Applications && echo XCODE_COPIED

# what we should have checked
du -sh /Applications/Xcode.app          # 3.8G, or it did not happen
/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -version

3. The permission that was reported granted

macOS gates screen recording behind a privilege that only a human or an MDM profile can grant. Apple's kickstart tool prints a clear warning when the privilege is missing. Our probe looked for that warning and, finding none, reported the privilege granted.

When the tool has no privilege at all it prints nothing. No warning, no anything. The absence of a warning looked exactly like success. The full story is in the Mac that logs you in and shows you nothing. The fix was to stop asking the tool's opinion and sample real pixels from the framebuffer instead.

4. The desktop that was reported dead

To confirm a user was logged in to the GUI, we checked who owned /dev/console. On a new machine it said root, so we concluded nobody was logged in. Then we checked a customer's machine, which had a user logged in for two days. It also said root.

For about ten minutes a paying customer's desktop was reported as down. It was fine. The device node ownership simply does not mean what we assumed. who lists a console line only when a session exists, and that is what our telemetry reports now.

5. The error that was a constant

A machine on a brand new macOS major would not auto-login. Apple's sysadminctl -autologin set returned SACSetAutoLoginPassword error:22. New OS, new error, obvious conclusion: the new release had broken scripted auto-login.

Then we ran the same command on a healthy machine one major behind, where auto-login had worked for weeks. Same error. The command fails identically on working and broken systems, so it carried no information about either. The real cause turned out to be a transient first-boot condition. The confident diagnosis was wrong, and the control disproved it in under a minute.

The rule, and two corollaries

Every one of the five fails the first question. Four days went into chasing them. None of the underlying systems were broken in the way the checks claimed.

What changed

The screen check samples the framebuffer and counts colours. The session check uses who and ships in telemetry, so a machine at the login window raises an alert instead of looking healthy. Copies are verified by size and by running the copied binary. And no failure gets a root cause until the same check has been run on something known to work. None of that is clever. All of it is the same question, asked before the check is written instead of after it has lied.