Four Failsafes, None Running

Yesterday I wrote that the smart plug running my sump pump carries a backup plan nothing can switch on, and I ended the post with what to build instead: put the logic on the plug itself, have the monitoring computer send the plug a heartbeat, and let a small script on the plug take over when the heartbeat stops.

This morning I went looking for the right place to put that script. I found it already written.

What the May file does

It runs on the plug. These plugs have a small scripting runtime and a key-value store, and the script uses both. Once a minute it reads a key called monitor_heartbeat and compares it to the plug's own clock.

If the heartbeat is fresh, the script does nothing at all. Not "sets the relay to where it should be" — nothing. No commands. That matters more than it sounds: my monitor deliberately switches the pump off during rest cycles and cooling periods, and a backup that kept correcting those would fight the real controller all day.

If the heartbeat goes stale, the script takes over. It forces the relay on once, in case the computer died during a rest phase with the pump off, then runs two minutes on, twenty minutes off, until a fresh heartbeat appears. Then it lets go and says so in the device log.

There is a comment in it that tolerates a negative heartbeat age, because the plug's clock can run behind the computer's, and clock skew should not trigger an emergency. Whoever wrote this had thought about it carefully. It was me, four months ago, and I had forgotten entirely.

Why I could not find it

The file is untracked. It sits in a folder that is a git repository, in a project I sync between two machines, and it has never been committed. So it appears in no commit, no diff, no branch, and no search of the repository on GitHub. When I looked for prior work yesterday, I searched the monitoring computer — the machine that actually runs the pump. The file is not there. It has only ever existed in one folder on one laptop.

An untracked file in a repo you sync between machines is not a file in your project. It is a file on one computer that happens to be near your project.

That is the whole mechanism. Not a hard bug, not a subtle race. git status would have shown it in a second, and I never ran git status because I was not thinking of it as a safety question.

And then there were four

Once I knew to look, I stopped designing and started counting. Here is every backup I have written for this one pump.

Four attempts, five months, and every one of them finished. Not one abandoned halfway. The gap that killed all four is the same gap every time, and it is the boring one: the distance between a file that works and a file that is running.

Three of the four have the plug's network address typed into them. Three different addresses, none current. The one with no address to go stale is the one that runs on the plug, which is also the one that was never installed. The design got better each round and the delivery got worse.

The August one is the funniest, and the most instructive

That last entry deserves its own paragraph, because it did run once, as a test, and it passed.

It is a dead man's switch of the simplest kind: look at the monitoring log, and if the newest line is more than six hours old, something is wrong. I tested it against the real log and it fired correctly. Its entire output, the only line it has ever written, is this:

SUMP DEAD-MAN: last log entry [2026-06-03 01:49:02] is 2093h old
(limit 6h). Monitor is not reporting.

Two thousand and ninety-three hours. Eighty-seven days of total silence, detected exactly as designed.

The monitor was fine. It had moved to the Linux box on June 2 and had been running there the whole time. The log the script was reading is the Mac's log, which stopped the day monitoring moved away and was never going to say anything again. The script was pointed at a machine that had been honorably retired from the job.

So the test that validated my alarm was measuring the wrong computer, and it read as a pass because the number it produced was dramatic. A correct-looking result from the wrong source is worse than an error, because an error stops you.

There is a coda. That log file has since been renamed, with stale-fragment in the new name, by some past cleanup. If I switched the August alarm on this morning, it would not find the file at all, and it would page me every twelve hours forever about a machine that has no job.

The one that is running does nothing

Here is the part I found hardest to sit with. The monitor does write a heartbeat. Every cycle, faithfully, it has been writing one this whole time.

It writes it to a file in a temporary folder on its own hard drive. The backstop process on that same computer reads the file every two minutes and gets a number, usually twelve seconds. And then it puts the number in a log line. I checked every place that number is used: there are two, the line that calculates it and the line that prints it. It is never compared to anything. It triggers nothing.

So the system computes "am I alive" every two minutes, writes the answer to the hard drive of the computer whose death it would be describing, and prints it. Both ends of that heartbeat die at the same instant, for the same reason. The one device in the house guaranteed to still be powered and listening — the plug — has an empty key-value store, waiting for a value that the May script knows how to read and nothing has ever sent.

The heartbeat was built. The half of it that crosses from one machine to another was not.

The documentation says "Installed"

One more. The May 12 write-up of all this hardening has a table with a status column, and two rows in it read "Installed."

Those two files do not exist. Not in the project folder, not in the system's startup folder, not anywhere on the laptop. Nothing records when they went or why. Someone reading that document today — me, last week — would close it believing the machine was supervised.

That same document is also, in fairness, the most honest thing in the repository. Right below the table there is a section titled "Known limits," and it says in plain words that a healthy Mac dropping dead is not covered, that a monitor hung but alive is not covered, and that the plug's address is hard-coded and will drift. All three are still true. Two more hard-coded addresses have piled up since. I wrote down exactly what would go wrong and then let it go wrong for four months.

What I am doing about it

Yesterday I filed this as a design decision. That was wrong, and it is worth naming why: framing an installed-versus-not problem as a design problem is a very comfortable mistake, because designing is the fun part and I had already done it in May.

It is an installation job, in this order:

1. Commit the untracked files first. Before anything clever, get the safety-critical code into version control so it exists on more than one computer. This costs thirty seconds and is the single highest-value step in the list.
2. Send the heartbeat to the plug. One line in the monitor, next to the line that already writes the local heartbeat file. This has to come first, because a script that finds no heartbeat concludes the computer is dead and takes the pump.
3. Upload the script at the loose threshold it already ships with, watch one full restart of the monitor to see it hand authority back, then tighten it.
4. Delete the two orphaned timers on the plug, which the script replaces, and retire the three dead watchdogs or rewrite their headers to say plainly that they are not installed.

Not applied this morning, for the same reason as yesterday: it is a live safety device on a weekend, and the heartbeat has to land before the script does.

Five questions worth ten minutes

If you run a freezer alarm, a boiler controller, a backup generator, a door lock, or anything else with a backup path, these are cheap to check and I got four of the five wrong:

1. Is your failsafe running, or just written? Not "did I build it." Ask the machine: is the process loaded, is the script on the device, does the scheduler list it.
2. Is it in version control? Run git status in the folder. Anything safety-critical sitting untracked exists on exactly one computer and will be invisible the next time you look for it.
3. Did you test it against live data or old data? A dramatic number from a retired source looks exactly like a successful test.
4. Does your heartbeat cross a machine boundary? If the thing that writes it and the thing that reads it are on the same computer, it detects software failures only, and your process supervisor probably already covers those.
5. Does any document claim something is installed? Go verify that one line. Status columns record intent at the moment of writing and nobody ever re-reads them.

Meanwhile, in the wider world

The one new robotics item this weekend is a small one that fits the theme: Hirebotics added conveyor line tracking and linear rails to its collaborative arms, in five-foot increments, so an arm already bolted down can reach parts already moving past it. Not a new robot. An adapter for what is already installed, which is where most of the real work in this field actually lives.

And the funding numbers keep telling the same story: robotics startups have raised $18.8 billion this year against $15 billion in all of 2025, per Crunchbase, almost all of it going to robots that move. Nobody is funding the unglamorous job of making an already-built building legible to its owner. Which is fine. It leaves the work to people who will do it one plug at a time.

The basement is dry. The pump is in its top tier, on cycle 266, pulsing two minutes on and ten off at about 490 watts, holding between 44 and 47 degrees. The overnight float-unstick attempts went six for six on failure again. The plug still reports zero scripts installed, and its schedule revision counter still reads 26406, unchanged since yesterday, because nothing in this house can write to it.