The One-Line Fix That Silenced 423 Alerts

This corrects two earlier posts: the one where my monitor went silent and the one where I discovered its heartbeat had been lying by telling the truth. Both blamed a Gmail rate limit for an ongoing blackout. The rate limit was real. It was also over before the blackout started, and I would have known that on day one if I had run a single command I kept putting off.

The float switch in my sump pit stuck at the end of July. Since then my software has run the pump on a timer — two minutes on, ten minutes off — and emailed me about every cycle. On August 20 it sent cycle number 3,509 and then stopped.

I spent three days working out why, from the outside. I read inboxes. I diffed timestamps. I counted gaps in a numbered sequence and built a theory about how the failure had progressed. I wrote two posts about it, each one confident and each one wrong in a different way.

The whole time, the answer was sitting in a log file on a machine in my house, and every one of those briefs ended with the same unfinished item: go read the monitor's log on the primary host. I carried it for four days. This morning I finally ran it. It took ninety seconds.

The Part I Got Right

First, the good news, which I want to state plainly before the rest of it: the rate limit theory was correct, and the log says so in the mail server's own words.

[2026-08-20 05:12:56] ERROR: Failed to send email (LOG): (550,
  b'5.4.5 Daily user sending limit exceeded. For more information
  on Gmail 5.4.5 sending limits go to ... - gsmtp')

There are twelve of those, one per cycle, running from 05:12:56 to 07:25:00. That is exactly the twelve-cycle block I had identified by inference — cycles 3,489 through 3,500 — without ever seeing an error message. One stuck float, emailing every twelve minutes for a month, had genuinely exhausted a Gmail account's daily sending allowance. A basement device DDoSed my inbox. That happened.

Then, at 07:27, the daily quota window rolled over and sending resumed on its own. Cycles 3,501 through 3,509 went out normally. The rate limit incident lasted two hours and fixed itself.

Which raises the obvious question. The blackout I have been writing about is seventy hours long and still going. If the quota problem ended at 07:27, what has been happening since?

The Part I Got Wrong

Here is the log, one minute after the last message any human received:

[09:15:21] LOG email sent to smart-home-monitor: ... cycle 3509 ...
[09:16:33] Shutting down monitor. Leaving plug in current state.
[09:16:33] === Sump Pump Monitor Started ===
...
[09:27:20] TIER_3 cycle 3510: pump ran (495.7W, 48.8C)
[09:27:20] LOG email skipped: no recipients configured

LOG email skipped: no recipients configured.

Four hundred and twenty-three times since.

The monitor was restarted at 09:16:33 and came back up with an empty recipient list. It has not attempted to send a routine notification since. Not failed to send — not attempted. Every twelve minutes for three days it has decided, correctly and by design, that it had nobody to talk to, written a note about it in a file nobody was reading, and gone back to running the pump.

The code that does this is not clever or broken:

NOTIFY_EMAILS_LOG = [e.strip() for e in
    os.environ.get("NOTIFY_EMAIL_LOG", "smart-home-monitor@...").split(",")
    if e.strip()]

An unset variable falls back to the default address. A variable set to the empty string beats the default, splits into a list containing one empty string, gets filtered down to an empty list, and quietly disables the channel forever. There is a real difference between "not configured" and "configured to nothing," and this line treats the second like the first.

And the configuration file tells you exactly when:

$ ls -la ~/home-assistant/.env
-rw------- 1 tdeshane tdeshane 1183 Aug 20 09:16 .env

$ diff .env.bak .env
49a50
> NOTIFY_EMAIL_LOG=

Modified at 09:16 — the same minute as the restart. One line, appended to the bottom of the file. It does not appear in any of the four backup copies. The service manager logged a clean stop and start, so this was a deliberate restart, not a crash.

Somebody with access to that machine turned off the noisy channel during the incident and never turned it back on. It is my machine and my file, so the honest version is that somebody acting as me killed the noise, it worked, and nobody wrote it down.

It Was a Good Fix

I want to be fair here, including to myself, because the instinct to call this the villain of the story is wrong.

It was the correct intervention. The routine channel was the thing burning the quota — a hundred and twenty messages a day about a condition that had not changed in a month. Emptying that one recipient list silences exactly that channel and nothing else. It is surgical.

And the surgery hit its target. My code sorts every message into URGENT or ROUTINE, and the urgent path was left completely untouched:

$ grep -c "URGENT email skipped" sump_pump_monitor.log
0

Zero. In the entire history of this system, not one urgent message has ever been suppressed. Whoever made this change knew what they were doing.

The Sixty-Two Messages That Should Not Have Been in That Bucket

Here is what the mute actually cost, and it is not the routine chatter.

[10:51:24] TEMP WARNING: 50.1C approaching limit (60.0C)
[10:51:24] LOG email skipped: no recipients configured

Sixty-two of those since the restart. My smart plug runs at 46 to 50 °C, and the emergency cutoff is 60 °C. Nothing here was an emergency. But "this component is sitting at eighty-three percent of its shutdown threshold" is not the same class of message as "cycle 3,510 completed normally," and my routing matrix cannot tell them apart. Both are labelled ROUTINE. So a switch aimed at status chatter took a thermal signal with it.

That was a latent classification bug. It had been sitting in my code for months doing no damage at all, and it was completely invisible until somebody used the mute switch. This is the ordinary way monitoring systems fail: not one big mistake, but a correct action landing on a wrong assumption that nobody had reason to examine.

The Correction I Care About Most

There is a third error, and it is the one I would most want a customer to understand, because it is about method rather than configuration.

Yesterday I reported that fifteen messages had gone missing before the blackout, and I built a whole narrative on the shape of those losses — intermittent drops, then a longer outage, then brief recovery, then death. I said that shape "is what a rate limit looks like from the inside."

I got those fifteen by looking for gaps in the cycle numbers. Here is what the log says about each one:

One real drop, twelve refusals, and two messages that were never missing at all — the system had simply sent something more interesting instead. I had assumed the cycle numbers formed a dense sequence in the email stream. They do not, and I had never checked, and my false assumption produced a beautifully suggestive pattern that happened to confirm the theory I already held.

Three days of careful outside-in reasoning, and every wrong turn came from inferring the behaviour of a system I could have simply asked.

Three Rules I Am Taking Out of This

1. Every mute needs an expiry and a banner

If your monitoring has any way to silence alerts — a maintenance mode, a snooze, an acknowledgement, a muted channel — then it needs three things: a default expiry, a visible marker wherever health is displayed, and a line in the daily summary.

My system sends a daily digest. It was delivered successfully on the 21st and the 22nd, both times while hundreds of messages were being discarded, and both times it said OVERALL: OK. It was telling the truth. The process was alive, the pump was cycling, everything it checked was genuinely fine. It just had no idea that a channel had been switched off. One extra line would have ended this on day one:

LOG channel   : MUTED (no recipients) since 08-20 09:16 — 70h
URGENT channel: todd.deshane@gmail.com

A monitoring system has to report its own configuration, not just its own state.

2. Suppression is not failure, and you cannot find it the same way

For three days I hunted for evidence of failure: bounces, error text, refused connections, retry storms. I found none for this outage, because nothing failed. The messages were never built.

This inverts the whole diagnostic approach. A failing system produces error logs, metrics, retries and bounces — you can find it by looking for damage. A suppressed system produces absolutely nothing, and can only be found by looking for absence. Almost all instrumentation, including mine, is built for the first case. My exception handler is well covered. The branch where the code decides not to act at all had no counter, no metric, and a log line marked informational.

3. Classify alerts by what you lose, not by how noisy they are

My urgency levels really encode "does this need to wake Todd up." That is the wrong axis. The right question is what you lose if a message is never delivered at all — and a temperature trending toward a cutoff has a very different answer than a routine cycle confirmation.

The rule I am adopting: a channel you might plausibly mute should contain only messages you are willing to lose entirely. If a message does not clear that bar, it belongs somewhere else. Anything in between needs a third tier that mute cannot touch.

What I Would Ask a Vendor

If you are paying someone to monitor a pump, a boiler, a walk-in cooler, or a building, these are worth asking. I know all four from the wrong side now.

• Can alerts be silenced, and what happens then? Every real system has a mute. Ask whether it expires on its own, and where it shows up when it is active. If the answer is "you'd just remember," that is my August.
• What does your dashboard say when a channel is switched off? If it still says everything is fine, it is reporting on the equipment and not on itself.
• Which alerts share a bucket? If a thermal warning and a routine status update travel together, one decision can take both.
• Where does the check for silence run? If it runs on the device or the gateway, it cannot detect the device being unable to reach you. It has to run somewhere else — on the receiving end, looking for arrivals that did not happen.

That last one is the fix I keep coming back to, and this incident settled it. A job on any other machine that alerts when the newest message from the pump is more than thirty minutes old is about five lines of code. It would have caught the rate limit on day one. It would also have caught the empty recipient list, a bad password, a dead network card, and whatever breaks next — because it tests whether the message arrived, and does not care why it did not. Two entirely different root causes in one incident, one check that catches both.

Meanwhile, in the Basement

None of this touched the pump. Cycle 3,855 ran this morning at 07:04, drawing 492.3 W at 48.8 °C, exactly on schedule. The plug reports 779 hours of uptime and zero reboots. Thirty-two days of a stuck float works out to roughly 128 hours of motor runtime, 64 kilowatt-hours, and about ten dollars of electricity.

A fifteen-dollar smart plug and a hundred lines of control logic have now run flawlessly through four separate failures of my monitoring software: a threshold I never measured, a watchdog that could not detect anything, a heartbeat that measured the wrong property, and a remediation that outlived its incident. Every single failure was in the layer I wrote, and not one of them was an AI problem. There is no model you can add to a system that has stopped talking and does not know it.

The cheapest item on my list is still a flashlight. It is the only instrument in this entire setup with a perfect record, and it is the only one I have not used.