My Loud Alerts Never Stopped. That's Why I Missed 1,474 Quiet Ones.

There is a good piece going around the buildings world this month arguing that predictive maintenance is being killed by alarm fatigue. Not by bad algorithms — by alerts nobody reads. The author describes watching an operator clear forty-one alerts in a couple of minutes without opening a single one.

That is a real problem and the diagnosis is right. Too many alerts and people stop reading them. The standard fix is to tier your alerts: urgent things go to a phone, routine things go to a log somewhere, and the operator's attention gets spent on the short list instead of the long one.

I tiered my alerts. I did it years ago and I still think it was correct.

Then I spent this morning finding out that one of my tiers had been switched off for eleven days, and that the reason I never noticed is that the other tier was working perfectly the entire time.

What I Actually Found

My sump pump monitor writes everything it does to a log. That log currently runs to 140,633 lines going back to February. This morning it contained 1,474 copies of this:

[2026-08-31 06:58:08] LOG email skipped: no recipients configured

The first one is timestamped August 20 at 09:27. The last one was eleven minutes before I went looking. They are still accumulating as I write this, roughly one every twelve minutes, which is how often the pump cycles.

Here is what was in the messages that never went anywhere: every per-cycle pump record, with wattage and motor temperature. Every time the plug switched on or off. And every attempt by the pump's self-repair routine to unstick a float that has been failing since April, along with whether it worked.

That last one matters. The self-repair routine is the most interesting thing my system does and the part I most want a record of. It was announcing itself into a void.

The Cause Was a Blank Line

The monitor decides who gets a routine, non-urgent message like this:

NOTIFY_EMAILS_LOG = [e.strip() for e in os.environ.get(
    "NOTIFY_EMAIL_LOG", "smart-home-monitor@agentmail.to"
).split(",") if e.strip()]

Read that and you would reasonably conclude the system cannot fail this way. There is a hardcoded default right there, and the default is correct — it is the actual address I want routine records sent to. Even if the config went missing entirely, the mail would still go to the right place.

Now here is line 50 of the config file:

NOTIFY_EMAIL_LOG=

The variable is set. It is set to nothing.

os.environ.get(key, default) gives you the default only when the key is absent. A key that is present and empty returns an empty string, and an empty string is a value, so the default never fires. "".split(",") is a list containing one empty string, the if e.strip() filter throws it away, and the recipient list is empty. Every routine message since then has been composed, addressed to nobody, and dropped.

The config file's modification time is August 20 at 09:16. The first dropped message is at 09:27. Eleven minutes.

My default was right. My config overrode it with nothing, and nothing beat it.

Blank Is Not Unset

I want to sit on this longer than the bug deserves, because it is not a Python quirk. It is a mismatch between two mental models that shows up everywhere.

The person editing the config thinks: I left that blank, so it will use the default.

The code thinks: the operator explicitly specified nothing, and who am I to argue.

Both are reasonable. Only one of them is what happens. And the same gap exists in essentially every configuration system a small building's stack touches — .env files, Docker Compose environment blocks, Kubernetes ConfigMaps, Home Assistant YAML, and any shell script where somebody wrote ${VAR-default} when they meant ${VAR:-default}. In all of them, an empty value silently wins against a carefully chosen fallback.

If you have ever half-filled in a config template — left a key in place with nothing after the equals sign, meaning to come back to it — you have probably done this. The system will not tell you. It has no way to distinguish your blank from your intention.

The Part That Should Worry You More

The bug is a one-line fix. The reason it survived eleven days is the interesting part, and it is not fixable with a line.

If you had asked me at any point during those eleven days whether my monitoring was sending email, I would have said yes, and I would have been able to prove it. Messages arrived. They were accurate. They were on time. They came from the right machine. Every check I could think to run came back clean, because every check I could think to run was really a check on the loudest channel.

That is the failure mode nobody puts in the alarm-fatigue writeups. Tiering your alerts is the correct response to alert overload, and I would still recommend it. But tiering has a cost that shows up much later:

A system that is completely dead is easy to catch. Everything stops, somebody notices within a day. A system that is partly dead is much harder, because the working half acts as a character witness for the broken half. The evidence you would use to detect the failure is being generated, correctly, by the component that did not fail.

This is worse than it sounds in buildings specifically, because the tier that usually breaks is the quiet one. The urgent path gets tested constantly — something goes wrong, the phone buzzes, you know it works. The routine path may go months without anyone deliberately confirming a message arrived. And the routine path is where the slow-moving data lives: the wattage creeping up, the temperature trending, the repair that ran and failed for the ninth time. All the things that would have told you about a failure before the urgent tier had to.

The Test, and It Takes Ten Minutes

In my last post I suggested unplugging your least important sensor and timing how long until something tells you. That test catches a dead sensor. It does not catch this.

Here is the one that does. Get a list of every distinct way your system can notify someone — every email address, every SMS number, every push channel, every webhook, every dashboard that is supposed to update. Most small buildings have between two and five. Then, for each one, answer a single question:

When did this specific channel last successfully deliver something?

Not "did anything alert me recently." Per channel. With a date.

If you cannot answer that for one of them, that channel is unverified, and unverified means you should assume it is off until proven otherwise. Mine was off for eleven days and I had no way to answer the question until I went and counted failures in a log by hand.

The fix that follows is unglamorous and cheap: each channel needs its own freshness check, and something has to complain when any one of them goes quiet for longer than its normal interval. Not one heartbeat for the system. One per path. If the routine channel normally delivers every twelve minutes and has delivered nothing in two hours, that is an alert, and it belongs in the tier that actually reaches a human.

It is the same principle as that post's, applied one level down. Sensors report problems; almost nothing reports its own absence. It turns out the alerting layer has exactly the same defect as the sensing layer, and for the same reason — because when the thing that reports failure fails, the way it tells you is by not telling you anything.

Why I'm Publishing the Bug

Because the honest version of building monitoring is that the hard part was never detection. Detecting a problem with a pump is arithmetic that was unremarkable in 1985, running on a sensor that now costs less than lunch. The hard part is the boring, unrewarded work of proving the plumbing is still connected, on a Tuesday, when nothing is wrong.

There is a widely cited figure, attributed to work at NASA and the US Navy, that only about 18% of equipment failures follow an age-related pattern. The other 82% are effectively random with respect to how old the machine is. That statistic is the entire argument for condition monitoring over a maintenance calendar, and it is the number I would put in front of any building owner still scheduling by date.

But it assumes something nobody checks. It assumes the monitoring is on. An 82% argument for continuous monitoring is worth precisely nothing if a tier of that monitoring has been quietly addressed to nobody since the twentieth of the month.

My pump is fine, by the way. It cycled at 06:58 this morning, 493.9 watts, 48.4 degrees, cycle 4,802. The system I built to watch it did its job the entire time. It just stopped telling half the story, and the half that kept talking sounded completely convincing.

Read more: Everyone publishes capability, nobody publishes uptime | My heartbeat said everything was fine | Smart building on a shoestring | Fifty-dollar vibration sensors

Sources: primary data from this author's sump pump monitor log (140,633 lines, February–August 2026) and its configuration; AutomatedBuildings, August 2026, on alarm fatigue in predictive maintenance, including the ~18%/82% failure-pattern figure attributed to NASA, the US Navy, and ARC Advisory Group.