Same Rain Forecast, Opposite Verdicts

At 00:40:47 last night my sump pump's guardian process sent an urgent notification. Subject: SUMP GUARDIAN: max run cutoff. Body: the pump had been running for 4.0 minutes against a 4.0-minute limit, and the relay had been "forced OFF for rest."

By the time that notification went out, the relay had been back on for a second.

The guardian cut the pump at 00:40:45. At 00:40:46 the main monitor noticed the plug was off, logged Plug output is OFF unexpectedly! Turning back ON., and turned it back on. At 00:40:47 the guardian's alert went out describing a cutoff that no longer existed. Forty seconds after that the monitor decided the pump had run too long after all, and cut it itself.

Five hours earlier the same thing happened at 20:21:21, except that time the pump finished its cycle normally, 48 seconds after the guardian tried to stop it. Total run: 4.7 minutes. Nothing was wrong except the page.

I have written about these two processes fighting over one relay before. What I had not done was ask why they fight when they do. The answer turned out to be a single function that both of them call, with opposite readings of what it means.

Two comments, one commit

Both processes ask the same question before deciding how worried to be about a long pump run: is it wet out? They share a small weather module. It returns wet if it is raining now, if at least 1 mm fell in the last 24 hours, or if 2 mm is forecast for the next 24.

Here is what the monitor does with the answer. This is the comment above its stuck-pump check:

# Dry weather: stay strict (3 min) because any sustained run is suspicious.
# Wet weather: relax (default 6 min) because legitimate heavy-rain
# cycles can pump for longer without being stuck.

And here is the guardian, on the same question:

"max_run",  # max-run while wet means we may actually be losing the flood battle

So when it rains, the monitor concludes a long run is more likely to be normal and gives it six minutes. The guardian concludes a long run is more likely to be an emergency, upgrades its alert to urgent, and cuts the pump at four minutes regardless of the weather.

Both comments are reasonable. A pump running long in the rain may just be keeping up. It may also be losing. Honestly, you cannot tell from runtime alone, which is its own lesson. But put them side by side and you have two safety systems that read the same fact in opposite directions.

They went in together, in one commit on the evening of June 2, when I moved the whole system onto a dedicated host. I wrote both comments. I never read them next to each other until this morning.

A backstop that fires first

The guardian exists as a backstop. It runs every two minutes, separately from the monitor, and its job is to catch the monitor failing. For that to work, its limit has to be looser than the monitor's. The monitor should always act first, and the guardian should only act if the monitor didn't.

Here is how the two limits actually line up:

In dry weather the nesting is right. In wet weather it flips. The backstop becomes a second controller that disagrees with the first, and the first one wins because it happens to poll faster.

It is slightly worse than the table shows. The guardian checks every two minutes and starts its clock the first time it sees the pump drawing power, so its "four minutes" is really somewhere between four and six. Last night's cut came 5.4 minutes after the pump actually started. The two limits don't just invert when it rains. They overlap.

What three months of logs say

I joined the guardian's log against the monitor's on timestamp, from the guardian's first cutoff on June 3 through this morning. There were 55 max-run cutoffs. I counted a cutoff as reversed if the monitor turned the relay back on within ten seconds, and split them by what mode the monitor was in at the time:

Two days ago I reported that the monitor reversed 23 of the guardian's first 52 cutoffs, about 44%, and I described that as a fight. It is a fight. But 44% made it sound random. When the monitor is on its own timer, the two agree every time. In normal mode the monitor overrode the guardian 25 times out of 26.

Of those 25, twenty-one have a weather verdict written down somewhere nearby, either the guardian's urgent escalation or the monitor's own "wet threshold" line. All twenty-one were wet, and none were dry. The other four had no verdict logged either way.

The part that matters for the person getting paged: the guardian has sent eleven urgent max-run notifications since June. The monitor undid ten of them within two seconds. The one that held came from the timer tier, where the two processes agree.

So which one is right?

It would be convenient to say the guardian is the bug. Sometimes it is. Seven of the reversed runs then ended on their own, at 3.9, 4.0, 4.0, 4.1, 4.7, 5.0 and 6.1 minutes. Those were normal pump cycles in wet weather. The guardian cut them and the monitor was right to put them back. Since September 1, the 90th percentile of normal runs is 4.1 minutes. The guardian's line sits right on it.

But the monitor is not simply right either. On September 13 it overrode the guardian five times in 28 minutes, the plug heated up about a degree per round, and the system tripped into an overtemperature lockout that left the pump without power for seven and a half hours overnight. Every one of those overrides was the same "legitimate heavy-rain cycle" logic that was correct at 20:21 last night.

So neither limit is really the problem. Nobody decided which one is in charge in the rain. Right now the answer is whichever process looks at the relay most recently, and that is a timing accident.

There is one more detail I keep coming back to. After each cutoff the guardian logs its rest countdown, and that line includes the relay state it just read. Of the 40 countdown lines it has ever written, 18 say output=True. The guardian has been writing down, in its own log, that the rest it is enforcing is not happening. It just never compares that reading to what it thinks it did.

Meanwhile, in robots

IEEE Spectrum ran a sponsored piece this week on safety for AI-driven robots. It is a sales pitch, but it collects real research. One result stood out. In the BadRobot study, researchers got a robot to say it refused a dangerous command while its motion controller went ahead and did it.

My version is milder and has no attacker in it. But the shape is the same: the layer a human reads says one thing and the actuator does another. My guardian's page says the pump is off. The relay says it is on. If you only read the pages, you'd believe the guardian.

The same week, an HCLTech and Omdia survey (summarized here) found that organizations still researching or piloting physical AI outnumber those with many systems in production by more than four to one: 81% against 19%. Facilities management was one of the top domains respondents expected to benefit, and predictive maintenance got called out by name. My guess is that a lot of those pilots stall on something like this: two reasonable components, each tested alone, deployed together in one commit and never checked against each other in the rain.

What I am changing

None of this is applied yet. In order:

1. Put the guardian above the monitor when it's wet. The guardian already asks the weather module whether it's wet; it just only uses the answer to choose the alert severity. It should use the same answer to choose its limit: four minutes dry (above the monitor's three), eight minutes wet (above the monitor's six). The stopgap is one line in the environment file, setting the limit to eight everywhere, which the guardian picks up on its next two-minute run with no restart. The cost is a slower backstop if the monitor dies on a dry day, which is why the real fix is the weather-aware one.
2. Alert on outcome, not intent. Five seconds after a cutoff, read the relay again. If it is back on, send "cutoff overridden by monitor" as a lower-priority note, not "relay forced off" as an urgent one.
3. Log the override from both sides. When the monitor turns on a relay it did not turn off, it should say the guardian probably did it. Right now each log only records its own success, which is why this took a timestamp join to find.

The check, if you run anything like this

1. List every component that can switch the actuator, and write each one's limit for each operating mode in one table. The backstop's limit must be the looser one in every row. If any row flips, that mode has two controllers and no backstop.
2. Find every input more than one safety component reads. Weather, occupancy, time of day, tariff. For each, write down which direction it moves each component. Opposite directions on the same input is a design decision someone should make on purpose.
3. Grep your alert templates for past-tense claims about the actuator. "Forced off," "shut down," "isolated." Each one should be backed by a read-back, not by the fact that the command was sent.

The basement is dry this morning. The pump ran five times between midnight and 06:30, all in normal mode. One of them got an urgent page that was false a second after it was sent. The smart plug's power-on default, which yesterday's post was about, still reads off.