Every Alert My System Ever Escalated Was About Itself

My sump pump monitor has two ways to tell me something. There is a quiet one, which goes to an archive mailbox I read when I feel like it, and there is a loud one, which goes to my personal email and fires a push notification to my phone. I built the loud one for emergencies.

It has fired four times since February. Here is the complete list.

[2026-06-18 13:06:05] URGENT: Sump pump: voltage LOW (brownout)
[2026-07-21 17:45:44] URGENT: SUMP PUMP: CRITICAL - Shelly offline 30+ minutes
[2026-07-21 18:59:34] URGENT: SUMP PUMP: CRITICAL - Shelly offline 30+ minutes
[2026-07-21 19:34:33] URGENT: SUMP PUMP: CRITICAL - Shelly offline 30+ minutes

One power company problem. Three instances of the smart plug dropping off my WiFi. Six and a half months of continuous operation, roughly 147,000 lines of log, and not one urgent alert has ever been about the pump.

I want to be precise about why that bothers me, because the obvious reading is the wrong one. It is not that nothing has gone wrong with the pump. Something is going wrong with the pump right now, and has been for five days.

What the Pump Is Doing While the Loud Channel Stays Quiet

The float switch in my sump basin has been stuck in the up position since Wednesday afternoon. Stuck up means the pump believes there is water to move, permanently. Left alone on its own wiring it would run until the motor burned out.

My monitor prevents that. It has taken the pump off the float's judgment entirely and put it on a timer: two minutes on, ten minutes off, repeating. That is the top rung of an escalation ladder the software climbs when it decides the float is unreliable. Here is where that stands as of this morning.

Five days. Five hundred and sixty-four consecutive cycles at roughly half a kilowatt, every single one confirming the pump is still being asked to work. Twenty-eight attempts by the software to jiggle the float loose, all of which failed honestly. And a perfectly silent phone.

This is, I should say, not a disaster. The basement is dry. The timer is doing the float's job and doing it adequately. But my system is running a hardware failure on a workaround, indefinitely, and it never told me. I found out because I go read my own logs for a weekly writeup. That is not a monitoring strategy. That is a hobby.

The Part I Assumed Was a Bug

My first assumption was that this was a delivery failure, and there is in fact one sitting right there. The quiet channel has a misconfigured recipient list. A blank line in a config file resolves to an empty string rather than the default I intended, so since August 20 the monitor has been logging this instead of sending anything:

[2026-09-07 07:05:51] LOG email skipped: no recipients configured

During this five-day lock it produced 631 notifications and skipped all 631. All time, 2,419 of them. Eighteen days of a monitor talking to nobody.

That is a real bug with a one-line fix, and I will get to it. But I want to be honest about what fixing it would have accomplished, because for about an hour I thought I had found the whole answer, and I had not.

If that channel had been working perfectly this entire time, those 631 messages would have landed in an archive mailbox alongside several thousand routine per-cycle status notes, formatted identically, sorted by arrival, indistinguishable. I would have found out about a five-day pump failure at exactly the speed I found out anyway: whenever I next went looking. The bug hid the paper trail. It is not what made me blind.

The Part That Was Working Exactly As Designed

So I went and read the notification code instead of guessing at it. There are exactly four places in the whole program that mark something urgent. Four. I read all of them.

Three of the four are watching the monitoring equipment, not the thing being monitored. Plug too hot, plug offline, power supply bad. Those are the conditions I decided were worth waking me up for, and what they have in common is that they are all cases where the monitor cannot see the pump.

The fourth one is the only genuine pump-condition alert in the system. It covers the float sticking down — the pump never running when it should. That is the opposite of what is happening. And it is conditional: it only escalates to urgent if the weather is wet, on the theory that a quiet pump in a dry week is just a dry week.

Which means the failure mode I have been staring at for five days, the float stuck up with the pump on a timer at maximum duty, does not appear anywhere on that list. There is no path from this condition to my phone. There never was.

And the code says so, out loud, in the escalation message itself. Here is the line the program sends every time it climbs a rung of the ladder:

This is designed escalation behavior, not urgent.

I wrote that. I remember the reasoning, and the reasoning was sound. A single escalation is routine. The ladder exists precisely so that a stuck float does not become an emergency — the machine handles it. Waking someone up for the thing your automation was built to absorb is how you train yourself to ignore your automation.

The flaw is not in that judgment. The flaw is that nothing in my system ever revisits it.

Urgency Decided Per-Event, In a World Where Duration Is the Signal

Every notification decision in that program is made about a single moment. This reading, this transition, this cycle. Nothing accumulates. There is no variable anywhere that represents "how long has this been abnormal," and so there is no way for the answer to that question to change the volume.

Cycle 1 in the top tier and cycle 564 in the top tier generate byte-identical handling. The first one is routine. The 564th is a hardware failure that has been papered over by a workaround for most of a week, and my software has no vocabulary for the difference.

Notice also what my escalation ladder actually escalates. Tier one is 20 seconds every 15 minutes. Tier two is 90 seconds every 10. Tier three is 120 seconds every 12. Each rung makes the pump work harder. Not one rung makes the alert louder. Five days into the top tier the system is working as hard as it is capable of working and speaking in precisely the same tone it used on the first cycle. Escalation and notification are two different axes, and I built a ladder on only one of them.

The Same Blind Spot Is in the Published Benchmarks

I went looking at what the predictive maintenance field says about alerting this week, partly to see how embarrassed I should be. The guidance is unusually specific right now, and worth quoting because the numbers are concrete:

• Teams seeing false-alert rates above 20% start ignoring notifications within about 30 days.
• A healthy program keeps response rates above 85%, false positives under 20%, and priority-one acknowledgment under 30 minutes.
• Multi-sensor correlation is credited with taking 500+ daily false alerts down to roughly 20.

All of that is sound advice about a real failure mode. Alarm floods kill monitoring programs; buildings buy these systems and go back to running reactive within a year, and alert fatigue is usually why.

But look carefully at what those three metrics have in common. False-positive rate, response rate, and acknowledgment time are all computed over alerts that were sent. Every one of them is a measurement of the noise you made. None of them is a measurement of the noise you failed to make.

Run my system against that scorecard for the past five days. Zero false positives across 564 cycles of a genuine equipment failure — a flawless rate. No missed acknowledgments, because there was nothing to acknowledge. No alert fatigue, because there were no alerts. My monitor scores perfectly on the entire published benchmark while sitting on top of a five-day failure it never mentioned.

The Table That Would Have Caught This

The fix here is not a better sensor and it is certainly not a model. It is a two-column table that I could have built in an afternoon at any point in the last six months, and did not.

List every failure mode you can name for the equipment. For each one, answer two questions: does the system detect it, and which notification tier does it reach. Then answer a third question that I would not have thought to ask a week ago: is that tier actually delivering right now.

Mine, filled in honestly, looks like this:

The two rows at the bottom are the ones that got me. Neither is exotic. Both are the kind of thing you only notice when you stop asking "what does my system detect" and start asking "what does my system detect and successfully tell somebody about." Those are very different questions and I had been answering the first one for six months while believing I had answered the second.

The row about the muted channel is worth its own note. A monitoring system that cannot monitor its own ability to notify you has a single point of failure sitting directly between it and the only thing it exists to do. Mine logged the problem faithfully, 2,419 times, to a file. Which nobody reads. Which is the point.

What I Am Changing

Two things, in order of how much they matter, which is the reverse of how obvious they are.

First, and less important: delete the blank line in the config, restart the service, and add a startup check that refuses to run with an empty recipient list rather than quietly logging about it 2,419 times. A monitor that starts up healthy with no way to reach anyone should treat that as a fatal error, not a footnote.

Second, and this is the real one: make duration a trigger in its own right. Any abnormal state that persists past some multiple of its expected lifetime gets promoted to the loud channel regardless of what tier originally classified it. My top tier is meant to be a temporary measure while I go fix a float. At 113.9 hours it is not a temporary measure, it is the new normal, and something should have said so out loud around hour six. Any threshold in that family — six hours, twelve, twenty-four — would have caught this days ago.

That second change is about eight lines of code. It requires no new hardware, no additional sensor, and no inference of any kind. It is the same shape as most of what I have found in this system over the past two weeks: the data was already there, written down, in a file I owned, and the gap was that nobody had asked it the right question.

The float, meanwhile, is still stuck. That one needs a flashlight and a basement, and no amount of software is going to change it. But at least now the software will mention it.