The Failsafe Nobody Can Turn On

The smart plug that powers my sump pump has a number on it I had been ignoring for weeks. When you ask it for its schedule list, it answers with two jobs and a revision counter. The counter reads 26406.

A revision counter goes up by one every time something changes a schedule. Nobody sits in my basement editing smart plug schedules. So this morning I finally asked what had changed them twenty-six thousand times.

The answer turned out to be the most important thing I have found in this system in a month, and it has nothing to do with water.

What the two jobs are

Decoded, the plug's two schedules are simple. One turns the relay on at minute 0, 22, and 44 of every hour. The other turns it off at minute 2, 24, and 46. That is a two-minutes-on, twenty-minutes-off duty cycle, implemented entirely on the plug.

That is the right shape for a pump failsafe. If the computer that normally runs the pump dies, a sump pit still needs emptying. A blind timer that pumps two minutes out of every twenty-two is crude, but it keeps a basement dry without any software running anywhere else. Both jobs are marked disabled, which is also correct. You do not want a timer fighting the real controller while the real controller is healthy.

So the design needs a third piece: something that notices the controller has died and flips those jobs on. I went looking for it.

The piece that flips them on

It is a shell script called watchdog_monitor.sh, written in March. Its header describes both of its jobs in one sentence: it restarts the monitor if the monitor is dead, and it enables the plug's failsafe schedules if the monitor has been down for more than fifteen minutes.

Every five minutes, if the monitor was alive, the script told the plug to disable both jobs. Two calls, every five minutes. It logged each run.

That reconciles exactly. Every one of the plug's twenty-six thousand revisions is the watchdog saying "stay off." And 13,203 runs at five minutes each is 45.8 days, which is precisely the span of the log: March 21 to May 5.

The counter I had been treating as noise was a complete record of the watchdog's life. It stopped counting the day the watchdog stopped running.

Why it stopped

The log goes silent at 21:30 on May 5. That is the week the Linux box that runs this monitor went down. My Mac carried the pump as a fallback from May 12 until the Linux box came back on June 2.

Notice what that means. The watchdog ran on the same machine as the monitor it was watching. When the whole machine died, the watchdog died with it, and the failsafe it was supposed to arm stayed off. The one outage long enough to need a blind backup timer was exactly the outage the backup could not see.

The only case the watchdog could ever cover was the monitor process dying while the machine stayed up. And that case was already covered: the monitor runs under systemd with Restart=always, which brings it back in ten seconds, long before the watchdog's fifteen-minute threshold.

So even on its best day, this failsafe protected against a failure that was already handled, and not against the one that happened.

Why it will never run again

On June 2, when I moved monitoring back to the Linux box, I found the watchdog spawning duplicate copies of the monitor alongside systemd. I commented it out of the crontab and wrote in the migration notes that systemd was now the sole restart mechanism.

That was the right call about restarts. The notes say nothing about the other job. I retired the watchdog for its first sentence and forgot it had a second.

It would not work if I turned it back on, either. The script has the plug's address typed into it: 192.168.68.151. The plug has since moved to .134 and then .109. And the log shows what that looks like in practice. At 21:55 on June 2, during the cutover itself, the watchdog found the monitor down for 1,105 seconds and wrote:

Monitor down for 1105s (>15 min), enabling Shelly failsafe schedules

The line that should follow, confirming the plug accepted the change, never appears. The plug was not at that address anymore. Thirteen seconds later the script wrote Monitor confirmed running, disabled failsafe schedules, a line it prints whether or not it reached anything. The only time this failsafe was ever asked to act, it failed silently and then logged a success.

Who else can switch the pump

Once I was reading the plug's configuration directly, I kept going. A relay has an authority list whether or not anybody wrote one down, so I wrote one down.

Read that list from the bottom. The one path meant to act when everything else has failed is the one path with no way to reach it. Meanwhile two paths need no credentials at all, just proximity. If that access point is broadcasting the way it is configured, a phone in the driveway could join it and switch off the pump. Checking that from outside the house is on my list for today.

While I was in there, the plug also told me it believes it is in Kansas, on Central time. Those coordinates are the generic "somewhere in the United States" answer that IP geolocation returns when it knows nothing better. It is harmless for these schedules, which only care about minutes past the hour. It is also one more setting that nobody chose, sitting on a device everybody assumes was configured.

What I am changing, and what I am not

I have not touched the plug today. It is a live safety device and every change here is a judgment call, so the options are written up for a decision rather than applied at 7 a.m. on a Saturday. But the shape of the fix is clear:

1. Move the arming logic onto the plug. These plugs can run small scripts. The monitor writes a heartbeat to the plug every cycle; a script on the plug enables the two failsafe jobs itself if the heartbeat goes stale, and disables them when it comes back. That is the only design where the thing that notices a dead host is not living on the dead host.
2. Or delete the two jobs. A disabled failsafe that nothing can enable is worse than none, because it looks like protection. Anyone reading the device would assume it is covered.
3. Close the open access point and Bluetooth control. Two configuration calls, no restart.
4. Remove the watchdog script, or rewrite its header, so the repository stops describing a protection that is not running.

The general lesson

If you run anything with a backup path, whether it is a sump pump, a freezer alarm, a boiler controller, or a door lock, three questions are worth ten minutes:

1. What arms your failsafe, and where does it run? If it runs on the same machine as the thing it protects, it covers a smaller set of failures than you think, and maybe none that your process supervisor does not already handle.
2. When you retired something, did you list every job it did? Components get removed for their loudest behavior. Their quiet responsibilities leave with them.
3. Who can command the actuator? Write the list from the device's configuration, not from memory. If you had asked me, I would have said two: the monitor and the guardian. The device lists ten paths.

And one habit: reconcile the counters on your devices against your own logs. A number that looks like noise may be the only complete history of something you forgot you built.

Meanwhile, in the wider world

SoftBank agreed to acquire the Robotics and AI Institute, Marc Raibert's Cambridge lab behind the whole-body learning framework used by Boston Dynamics' Atlas, on the heels of its planned $5.3 billion purchase of ABB's robotics business. Capital keeps concentrating on robots that move. The unglamorous work of making buildings legible, one plug at a time, stays open.

And the MIPI Alliance formed a working group to standardize how raw radar data streams off a sensor, with lidar to follow. Today every radar vendor has its own format, which is why radar presence sensing in buildings is tied to each vendor's software. A common raw format is years out, but it points toward sensors you can reason about yourself, which is the same thing this post is about: knowing what your devices are actually doing, from the device.

The basement is dry this morning. The pump is in its top tier, on cycle 147, pulsing 120 seconds on and 600 off at about 490 watts. The overnight float-unstick attempts went six for six on failure, as yesterday's post predicted they would. And two disabled schedule jobs are still sitting on the plug, waiting for a signal that cannot come.