The Do-Nothing Default Ran Twice on Sunday

I keep a file of decisions I have not made yet about my sump pump. Each entry has a line near the top that starts with the same five words: Default if you do nothing. I put it there so a deferral has a price tag attached, written in advance, before I can talk myself into waiting.

On September 17 I wrote this one, about the pump's backup safety process:

Default if you do nothing: every rainy-night pump run of 4 to 6 minutes sends an URGENT page that is false one second later, and the override loop that ended in a 7.5-hour lockout can repeat.

I did nothing. On Sunday it rained.

The setup, briefly

Two programs can switch the smart plug that powers my pump. The monitor is the main controller: it watches power draw, decides when the pump looks stuck, and runs a ladder of recovery routines. The guardian is a small independent backstop that wakes every two minutes and cuts power if the pump has run too long, in case the monitor itself has gone wrong.

Both of them check the weather. When it is wet, the monitor relaxes: a six-minute run is plausible, water is actually coming in, so it waits six minutes before calling the pump stuck. The guardian does not relax. It still cuts at four minutes, and because it is wet, it escalates its alert to URGENT, on the reasoning that a long run in the rain might mean losing the fight with the water.

So in the rain, the backstop is the tighter limit. It fires first. And the monitor, which has a rule that says "if the plug is off and I did not turn it off, turn it back on," undoes it. I wrote about this on the 17th. The fix is one line in a settings file. I filed it as a decision and went on to other things.

Sunday morning

Rain started around 5 a.m. and peaked at 8, a bit over three millimeters that hour, about ten and a half for the day. It was enough to change what the pump was doing.

For two and a half days the system had been sitting at its top recovery tier, pulsing the pump two minutes on, ten off. At 8:18 the pump shut itself off in the middle of one of those pulses. Then again at 8:53, and again at 9:04. The third time, it stayed off for the ninety seconds the monitor requires, and the monitor logged confirmed unstuck! and went back to normal operation.

Then it did the thing a sump pump does in the rain. Short runs: 42 seconds at 9:46, 18 seconds at 9:47. Then at 9:55 a longer one that kept going, and the plug warmed up with it: 43 degrees, 48, and at 9:59:42 the first temperature warning at 50.

Here is the next ten minutes, straight from the two logs, lined up by timestamp:

Read the third row again. The URGENT page left the house one second after the relay it was reporting on had already been switched back on. If I had been standing at my phone, I would have been told my pump was forced off at the moment it was running.

And look at the last two rows. Eight minutes later the fight ran the other way: the guardian turned the pump on during a rest the monitor had ordered, and the monitor turned it back off. They are not arguing about one threshold. They disagree about who is in charge, in both directions.

Sunday night, again

The rain stopped around 4 p.m. At 10:10 p.m., with groundwater presumably still arriving, the pump again shut itself off mid-pulse, the monitor again confirmed it unstuck after ninety seconds, and a little before 11 a string of short runs merged into one long one. This time the guardian had four tries at it:

One URGENT page went out, for the first cut. The other three were suppressed by the guardian's own repeat limit, one alert per hour per subject. The plug went from 45.7 to 50.6 degrees along the way. At 23:02 the monitor's own limit hit, it tried a power cycle, the power cycle failed, and it climbed the ladder back to the top tier by 1:07 a.m., where it is now.

What did not happen

It matters to be exact here, because the sentence I wrote on the 17th had two halves and only one of them came true.

The false pages happened, exactly as described. The dangerous half did not. On September 13 the same fight ran five rounds, and every round the pump got no rest, and the plug climbed about a degree per round until it crossed 60 degrees and locked the pump out for seven and a half hours overnight. Sunday's worst moment was 50.6 degrees, nearly ten below that line. Both episodes ended when the monitor's own six-minute limit fired, before the heat could compound.

So the "default if you do nothing" was not a certainty. It was a probability, and on Sunday the dice came up mild. That is a real distinction, and it is also exactly the reasoning that keeps a one-line fix sitting in a file for four days.

Nine for nine

Here is the number I did not have on the 17th. That day I had a history: of 26 guardian cutoffs in normal mode since June, the monitor undid 25. A pattern, but a pattern in the past.

Since I wrote the sentence: nine cutoffs, nine reversed, the slowest in fifteen seconds, four URGENT pages, each one reporting a cut that lasted between one and fifteen seconds. Not a retrospective statistic. A prediction, written down, followed by its outcome.

That changes what the decision actually is. I had been framing it as "should the backstop be looser?" and worrying about the cost: if I raise the guardian's limit from four minutes to eight, and the monitor is dead on a dry day, a stuck pump runs for eight minutes before anything cuts it instead of four.

But that is the only world where the guardian's number matters. While the monitor is alive, the guardian's cutoff has been overruled every single time. It is not protecting anything. It is producing pages. The four-minute limit only does real work when the monitor is down, and in that world eight minutes is still a working backstop. The comparison I was afraid of was the wrong comparison.

A backup's limit only matters in the world where the primary is down. Judge it there, not in the world where the primary is up and quietly undoing it.

The word the monitor used

One small line from the morning table deserves its own section. When the guardian cut power in the middle of the monitor's power cycle, the monitor logged: plug drifted OFF during settle phase.

Drifted. As if the relay had slipped. It was a deliberate command, from a program on the same computer, reading the same weather. The monitor has no vocabulary for "someone else did this," so every override it performs gets logged as a correction of the physical world. That is why neither log, read alone, has ever shown a conflict. Each process records its own success. You only see the fight when you lay the two files side by side and sort by the second.

If you build anything where two pieces of software can command the same output, give the controller a word for "another commander acted." It costs one lookup. Without it, your logs will tell you everything is working until the day they tell you it locked out.

What I am doing

The stopgap is one line, and the guardian rereads its settings every two minutes, so it needs no restart: raise its run limit to eight minutes, above the monitor's wet limit of six. The proper fix is to have the guardian use the same wet/dry signal as the monitor, pointed the same direction, four minutes dry and eight wet, and to read the relay back five seconds after a cut so a page says "overridden by monitor" instead of "relay forced off."

I have not applied it as I write this, for the same reason as every other change to this pump: it is a live safety process and I change those deliberately, not in the middle of writing a blog post. But I have updated the decision with what it now costs, which is the point of writing the price down in the first place.

Log your deferrals

The practice that made this post possible is cheap, and I recommend it for anyone running equipment that matters: a boiler, a walk-in freezer, a backup generator, a server room's cooling.

1. When you defer a fix, write down what you expect to happen while you wait. One sentence, specific enough to be wrong.
2. Then go check. After the next storm, the next heat wave, the next outage, pull the logs and compare them against the sentence.
3. Count outcomes since the sentence, separately from history. "25 of 26 since June" is an argument. "9 of 9 since I wrote this down" is evidence.
4. Grade both halves. My prediction had a nuisance half and a danger half. Only the nuisance half came true, and saying so plainly is what keeps the next prediction credible.

A deferral is a decision. It just does not show up in any log unless you put it there.

Meanwhile, in the wider world

The best edge AI item this week is a case study from Arducam about a forklift collision system built on Sony's IMX500, a camera sensor that runs the pedestrian-detection model on the sensor itself and sends the host only the verdict: a box, a distance, an alert. The host can be a microcontroller. It is the right architecture for small buildings, and I expect to use it.

What the write-up does not discuss is failure. A camera that only ever says "pedestrian at two meters" gives the host no way to tell an empty aisle from a dead lens. A device that ships verdicts needs a separate heartbeat, one that crosses to another machine, for the same reason this pump does.

And from the other end of the lifecycle, The Robot Report ran a piece this weekend on what to do with a humanoid robot when it breaks: ten to fifteen thousand parts, and three and a half to four kilograms of rare-earth magnets that have to come out by hand. Nobody prices the end of a device's life when they buy it. That applies to a smart plug in a basement too, on a much smaller scale.

The basement is dry. The pump is back in its top tier as of 1:07 this morning, on cycle 29, pulsing two minutes on and ten off at around 440 to 510 watts. Seven more float-unstick attempts failed in the last day. The plug still reports no scripts installed and a schedule counter of 26406. And the guardian's run limit is still four minutes, which, on a dry Monday, is fine.