My Pump's Backstop Was Independent Until the Outage Lasted 15 Minutes

IEEE Spectrum ran a sponsored piece this week from VicOne, a security firm that tests robots, with a title that stuck with me: "Your Robot's Safety Functions Already Work. What If the Input Lies?" Their examples are dramatic: a poster that a robot dog read as an instruction, an injected network message that moved a robot that was supposed to stand still, sound tuned to a gyroscope's resonant frequency. But the question at the center of it is plain, and it applies to anything with a safety layer. When the primary system gets bad information, does your protection detect the danger on its own, or does it depend on the same information?

I have a system with exactly that structure in my basement, and five months of logs from both halves of it. So I checked.

The setup: two programs, one plug

My sump pump plugs into a smart plug that can measure power and cut the outlet. A monitor program polls the plug every 30 seconds and decides when the pump runs. In June I added a second program I called the guardian, described in its own source as an "independent safety backstop." It runs on its own two-minute timer, with its own thresholds, and its job is to catch the monitor when it gets something wrong: a pump running too long, a plug that has wandered to a new address, a plug that has gone silent.

Separate code, separate schedule, separate alarm. It felt independent when I wrote it. Then I listed what it reads:

• the same smart plug, for whether the pump is running and how hot the plug is,
• over the same Wi-Fi, from the same computer,
• starting from a copy of the monitor's own state file.

Cross out everything the monitor also reads, and what is left is the timer and the thresholds. That is not nothing, but it is not a second set of eyes.

The test: when the monitor was blind, could the guardian see?

Every time the monitor fails to reach the plug, it writes a line. Every two minutes the guardian writes whether it could reach the plug. Since the guardian's first log line on June 2, the monitor has lost contact with the plug 329 separate times, for 37.7 hours in total. For each of those windows I checked what the guardian saw while it lasted:

For short outages, the guardian earns its name. Most of those are a single request the plug refused or dropped. The guardian asks at a different moment, gets an answer, and has a real reading while the monitor is retrying.

For long outages, it is a copy. Six times since June the monitor lost the plug for fifteen minutes or more, and in those six windows the guardian took 701 readings and got two, both in the last four minutes of the first one. Those six windows are 63 percent of all the time the monitor was blind.

Why the long ones are different

A short outage is usually one request failing. A long outage is usually something both programs depend on failing. Here are the six, grouped by day:

• June 23, 2.5 hours. Neither program could find the plug on the network, and the computer they both run on had lost name resolution. The guardian noticed and tried to page me twice. Both pages failed with the same network error, because they had to leave from the same machine.
• July 21, 2.3 hours across three windows. The plug stopped answering anyone, and came back only after a power-on reset at the outlet. (I wrote about those resets earlier this month.) Whatever was wrong was in the plug or its outlet, which both programs share.
• September 27, 77 minutes. The window in which the pump ran about 55 minutes with nobody watching.
• October 5, 17.8 hours. The computer lost its connection to the network segment the plug lives on, while keeping its internet connection. The plug never moved.

A plug that stopped answering, a confused network, a computer that lost a link. None of those is a monitor bug, and none of them can be caught by a second program that shares the plug, the network and the computer. In safety engineering this has a name, common-cause failure, and the fix has a name too: diversity. Different sensor, different path, different power.

What the backstop actually contributed

Here is the fair version. The guardian could not see during long outages, but it could tell me it couldn't see, and it did that well. On July 21, September 27 and October 5 its "plug unreachable" page reached my phone 12 to 13 minutes before the monitor's own "offline 30+ minutes" page, because its threshold is 15 minutes instead of 30. On October 5 it kept paging every hour, eighteen times, while the monitor sent once and went quiet.

But notice what that is. It is the same alarm, sent sooner and louder. Both programs were paging about the same symptom, seen through the same plug, sent from the same computer. And on June 23 the shared part was the part that broke: the guardian tried at 8:26 and 9:26, the monitor tried at 8:39, and all three pages failed with the same name-resolution error, because they all had to leave by the network the computer had lost. On the worst morning of the four, two independent programs produced zero pages between them.

There is a quieter version of VicOne's point in here too. Both programs decide whether the pump is running from the same number, the watts the plug reports. If the plug ever reports zero while the motor runs, both programs believe it at the same time. The guardian's "pump ran too long" cutoff cannot fire on a run it cannot see. The input doesn't have to be malicious to lie; it only has to be shared.

The fix is boring on purpose

None of this needs more AI. A smarter model reading the plug's watts is still reading the plug's watts. What it needs is one input that shares nothing with the plug:

• A second sensor on a second path. A battery water sensor at the rim of the pit, on a different radio than Wi-Fi, with its own alert. If water reaches the rim, it doesn't matter what the plug, the router or the computer thinks.
• One check that lives somewhere else. A heartbeat from the monitor computer to an outside service that pages me when the heartbeats stop. It is the one alarm that still works when the computer, the router or DNS is what broke, which is exactly the June 23 case.

Together that is well under fifty dollars and an afternoon. I am adding both this week, and lifting the pit lid while I'm down there.

The independence test, for any building system

1. List every input the backstop reads. Sensors, network, power, the computer it runs on, any file it copies from the primary.
2. Cross out the ones the primary also reads. What is left is the backstop's real independence. If nothing is left, you have a second opinion from the same witness.
3. Measure it on the long outages. Join the two logs: for every window where the primary was blind, did the backstop get a good reading? Bucket by duration. Independence during one-minute blips is easy. Independence during a two-hour outage is the whole reason a backstop exists.