My Flood Alarm Fired 17 Times While the Pump Was Running

At 2:09 a.m. on September 16, the smart assessor watching my sump pump wrote this line to its log:

FLOOD_RISK WARN | 495W 44.5C ON state=NORMAL | No pump run seen in 28.3h

Read it left to right. The plug reports 495 watts, and the relay is on, so the pump is running right now. The verdict says the pump hasn't run in 28 hours. The main monitor, reading the same plug, had logged "Pump started running" 51 seconds earlier.

That wasn't a single glitch. Over September the assessor raised the same flood warning 195 times. In every case the pump had run inside the window the warning said was empty. Seventeen of those warnings reached my inbox.

What the assessor is for

My sump system has three layers: a main monitor that runs the pump, a guardian that acts as a safety backstop, and a smart assessor that runs every 15 minutes and writes a one-line verdict in plain English. The assessor's job is to be the readable layer, the one that tells a person what's going on. Its WARN-level verdicts are emailed and pushed to my phone, with a three-hour cooldown per alert type.

FLOOD_RISK is the alarm I'd least like to miss. It means the pump hasn't run in 12 hours. If the pit is dry that's fine, but if the float is stuck off, the basement floods. With only a power sensor you can't tell those apart, so the alert's suggested action is sensible: "Eyeball the pit."

Two dialects, one parser

The assessor doesn't read the pump. It reads the last 400 lines of the main monitor's log and searches for runs. Here is the bug: the monitor describes a pump run differently depending on its mode.

The parser was written while the system spent nearly all its time in the timed tiers, so it learned that dialect. In NORMAL, the mode where the pump behaves like an ordinary sump pump, it sees the stop lines and files them as evidence that the float works. It never connects them to the pump having run. Once the system has been in NORMAL for 12 hours, the newest "run" the assessor can find is the last timed pulse, and it reports a flood risk.

So the alarm fires when things are going well. It goes off after the system has been off the timer and under normal float control for half a day.

The count

I joined every FLOOD_RISK verdict against the monitor's log and asked whether the monitor recorded a pump start inside the window the verdict claimed was empty.

Every one of the 195 was false. (Five more fired on June 2, the night I moved the system to new hardware and the log had a four-week hole, so I've left those out.)

The emails were worse than the log. Because of the three-hour cooldown, each episode arrived as a series: "No pump run seen in 12.0h," then 15.0h, 18.0h, 21.0h, 24.0h, 27.0h. On September 17 it reached 33.1 hours. Read in order, the subject lines describe a worsening outage. Meanwhile the pump ran 38 times in the second episode alone, about 44 minutes of logged pumping. It rained 0.0 millimeters on the 15th and 16th.

The channel that works carried the false alarms

This is the part that bothers me most. A few weeks ago I found that my monitor's lower-priority alert tier had no recipients: a blank line in a config file sent every message nowhere, while the log said "sent." The assessor doesn't have that problem. It has its own correctly configured recipient, and it delivers every time.

Over the last 60 days it emailed me 28 times. Eleven were "Shelly unreachable," and those were real events. Seventeen were FLOOD_RISK, and all of those were false. In my house, the most reliable alert channel was wrong 61% of the time, while the channel that would carry the real low-priority warnings delivered nothing at all.

The usual result of a channel like that is that you learn to ignore it. Then one day the pump really does stop for 12 hours, the email says what it always says, and you ignore that one too.

I had already called this field healthy

I should own this part. Every week I run a liveness audit that counts the distinct values of each field the system computes and flags any field stuck on a single value. For weeks the assessor's verdict was always the same class, HIGH_INFLOW, and I wrote about why that constant meant nothing.

On September 25 the audit showed that field taking three distinct values, and my brief that day described it as a constant field coming back to life. The new value was FLOOD_RISK. All of those verdicts were false.

My audit asked two questions: does the field vary, and could it take another value at all? It needs a third: when the value changes, does it agree with an independent record? A field can change without telling you anything new.

The fix was already in the function

The assessor already reads the plug's cumulative energy counter, the total watt-hours it has ever passed. It uses that counter to estimate "minutes of pumping per hour." It computes that number a few lines before the flood check, and the flood check ignores it.

Checking whether the pump has run in 12 hours doesn't require parsing anyone's log. It only requires checking whether the energy counter has moved. A counter doesn't have dialects, and it doesn't care which mode the controller is in. It's the same fix as last week's, where a two-minute sampler merged two pump runs into one: a cumulative total records what happened between your readings and outside your parser's vocabulary.

The fix is two changes: base the flood check on energy used, and teach the parser that "Pump started running" is a run. The assessor runs from a timer, so the fix doesn't even need a service restart. It hasn't been applied yet. The next time the system spends 12 hours under normal float control, the alarm will fire again.

One more thing. Every one of those 17 emails said "Eyeball the pit." That was the right advice, given for the wrong reason. It's also the same inspection I've had on my to-do list since February.

For your building

Many building monitors I look at are layered like mine: a controller, something that reads the controller's output, and something that writes the alert. Here's how I'd check yours:

1. Find every alert that reads another program's log or output. List each format the producer can write, including mode-specific ones, and check that the consumer understands all of them.
2. Prefer counters to parsing. If the question is "did X happen in the last N hours," a run-hour counter, an energy meter or a cycle count answers it without depending on anyone's wording.
3. Grade alerts against an independent record, not against your inbox. Join each alert to a second source and count how many were true. "It fired" and "it was delivered" both say nothing about whether it was right.
4. Treat a field that suddenly starts varying as something to verify. The new value may be the first true thing the field has said, or a new way of being wrong.