My Backstop Counted Two Pump Runs as One and Cut the Power

At 1:08 a.m. on Tuesday, my sump pump did exactly what a healthy sump pump does. The float rose, the pump ran for one minute, the pit emptied, and the float dropped. Ninety-one seconds later the float rose again and the pump started again.

Two minutes after that, the safety process I call the guardian cut the power. Its log line said the pump had been running continuously for 4.0 minutes, which is its limit.

The pump had been running for 2.2 minutes. Before that it had been off for a minute and a half. The guardian never saw the gap, because it only looks every two minutes, and both times it looked, the pump happened to be running.

That one wrong number started a chain that has kept my pump on a fixed timer for more than four days.

How a backstop measures "continuous"

The guardian is meant to be the simple, independent layer under my main monitor. A systemd timer runs it every 120 seconds. Each run reads the smart plug once: is the relay on, and is it drawing more than 100 watts? If so, and no run is in progress, it writes down "running since now." It erases that timestamp only when a later reading shows the pump idle. When "running since" is four minutes old, it cuts the relay.

That logic has a blind spot built in. If the pump stops and restarts between two readings, the guardian can't know. Two separate runs and an idle gap become one long run.

The main monitor reads the same plug every 30 seconds, so it has the real timeline. Here are both logs from Tuesday morning:

Of the 242 seconds the guardian counted as a single run, the pump drew power for about 151, roughly 60%.

What it cost

On its own, the first cut was harmless. The main monitor reversed it within a second, as it does with most of the guardian's cuts. (That fight between the two processes is its own story.)

The second cut is the expensive one. By then the main monitor had started its own recovery test: turn the pump off for 10 seconds, turn it back on, and check whether it's still pumping a minute later. The guardian cut the relay inside that window, using the same clock it had never reset. The monitor turned the pump back on, restarted the test, and 61 seconds later found the pump drawing 501 watts. It decided the float was stuck and dropped into TIER_3, its most cautious mode: two minutes on, ten minutes off, around the clock, no matter what the float says.

As I write this, it's still there: four days and six hours, 507 timed pulses, and 26 automatic "unstick" attempts with zero successes. Rain over those four days totaled 1.8 millimeters.

I'll be careful about the counterfactual. I can't tell you the recovery test would have passed if the guardian hadn't cut in. The pit was clearly taking on water that night. What I can tell you is that the test my system acted on was measuring a pump that had been restarted 61 seconds earlier, by a cut based on a run that never happened.

Not a one-off

I went back through every max-run cut the guardian has ever made, 75 since June. For each one I took the window the guardian counted as continuous and asked whether the main monitor had logged the pump stopping inside it.

Nine of the 75 cuts (12%) were made against runs the pump had already ended. They group into three nights: June 15, September 20, and this Tuesday. At the first cut on each of those nights, the real continuous run was under two and a half minutes against a four-minute limit. All three nights left normal float control within four minutes of that first cut.

That's a lower bound. The main monitor only writes start and stop lines in normal mode, so a merged run in any other mode leaves no evidence to count.

How often could this happen? Across 244 idle gaps logged between float runs, 20 (8%) were shorter than the guardian's two-minute interval. Any of them can disappear between two readings. Whether it does depends on where the readings land.

The fix is a counter the plug already keeps

You could make the guardian poll faster, but that only shrinks the blind spot. A better fix is to stop treating two readings as proof of what happened between them.

The plug keeps a cumulative energy counter, total watt-hours since it was installed, plus per-minute energy for the last three minutes. The guardian could already read it. Before cutting, it should check that the energy used since "running since" is close to what continuous running would use: at least about 90% of power times elapsed time. On Tuesday the real figure was about 60%. That check would have reset the clock, left the pump alone, and never touched the monitor's recovery test.

A counter like that can't miss the gap, because it doesn't depend on when you look. It's the same thing that saved me the night the plug dropped off the network mid-run: snapshots lie about what happened between them, and cumulative totals don't.

For your building

Almost every building monitor I look at has a rule built on a duration, like "compressor ran longer than X" or "door open longer than Y," computed from periodic readings. Here's how I'd check yours:

1. Find the sampling interval behind every duration rule. If the thing being timed can switch off and back on faster than that interval, the rule can merge separate events into one long one.
2. Compare the shortest real off-gap to the poll interval. Mine were as short as 91 seconds against a 120-second poll. If you don't know your shortest gap, measure it with a faster logger for a day.
3. Cross-check durations against a cumulative counter. Energy meters, run-hour counters, and cycle counters all record what happened between your readings. If the device has one, a duration rule that ignores it is guessing.
4. Count your backstop's actions against the fast log, not its own log. My guardian's log is perfectly consistent with itself. Only joining it to a faster source showed that 12% of its cuts were against runs that had already ended.