Two of Nine Emergency Alerts Died of the Emergency

The loudest thing my sump pump monitor can say is that it has lost contact with the pump. Thirty minutes of silence from the smart plug and it sends an email to my personal address and fires a push notification to my phone. The body of the message does not hedge:

The Shelly plug has been unreachable for 30 minutes. The pump is completely unprotected.

That alert has fired nine times since February. Yesterday I wrote about which conditions reach that tier at all. Today I went one question further and checked something I had never checked in six and a half months of running this thing: when the loud alert fires, does the message actually leave the building?

Seven times out of nine, yes. Twice, no. And the two failures are not random.

The Delivery Record

Here is every time the critical alert has fired, and what happened immediately after.

A 22% loss rate on the highest-severity alert in the system is bad on its own. But the loss rate is not the finding. The finding is in the error messages.

[2026-03-09 12:07:02] SHELLY CRITICAL: unreachable for 30 min
[2026-03-09 12:07:02] ERROR: Failed to send email alert: [Errno -3] Temporary failure in name resolution
[2026-03-09 12:07:02] ERROR: Failed to send ntfy alert: [Errno -3] Temporary failure in name resolution

[2026-06-23 08:39:03] SHELLY CRITICAL: unreachable for 30 min
[2026-06-23 08:39:23] ERROR: Failed to send email (URGENT): [Errno -3] Temporary failure in name resolution
[2026-06-23 08:39:43] ERROR: Failed to send ntfy alert: [Errno -3] Temporary failure in name resolution

Read those three lines in order. The monitor could not reach the plug. It tried to tell me it could not reach the plug. It could not reach the mail server, and it could not reach the push service, for the same reason it could not reach the plug. The host had lost name resolution. The network was down.

The alert and the fault were not two events. They were one event, observed twice, and the second observation could not travel because of the first.

My Redundancy Was Decorative

I have two notification channels specifically so that one of them can fail. Email through Gmail, push through ntfy.sh. Two different companies, two different protocols, two different phones' worth of notification surface. It felt like belt and suspenders.

Both of them need working DNS and a working uplink from the same small computer in my basement. On March 9 they failed in the same second. On June 23 they failed twenty seconds apart. Two channels, one failure domain, one point of failure wearing two logos.

The full log makes the pattern clean. Across six and a half months there have been 37 email send failures. Six of them were name resolution — my own network. The other 31 were remote-side problems: Gmail returning a temporary 421, a read timing out. There have been 6 push failures, and 5 of those were name resolution.

The remote-side failures are exactly what redundancy is for, and it works: Gmail hiccups, the push goes through. But those are the uncorrelated failures, the ones that were never going to hurt me much. The correlated failures — the ones where my network is the thing that broke — take out both channels at once, every time, and those are the failures that coincide with the pump being unmonitored.

I built redundancy against the failure I was not worried about.

The Dependency Table Nobody Fills In

The way to see this before it happens is to list what each path depends on and look for the overlap. Here is mine.

Three rows are shared by all three columns. Anything that breaks host power, the host's network interface, or DNS takes out the monitoring and both ways of reporting that the monitoring is out. The only faults my redundancy genuinely covers are in the last row, where Gmail and ntfy are independent of each other.

That is one row of real protection and three rows of shared exposure, and I would have told you before this week that I had two channels.

The Four Lines That Turned a Dropped Message Into a Blind Spot

A lost message is survivable if the system tries again. Mine does not, and the reason is the least interesting code in the file.

The function that sends notifications wraps both channels in try/except, logs the exception, and returns nothing. It also returns nothing on success. It has no return value at all, which means success and failure are literally indistinguishable to all 22 places that call it.

Then, in the function that watches for the plug going silent:

if not sm.shelly_critical_alerted:
    log(f"SHELLY CRITICAL: unreachable for {elapsed / 60:.0f} min")
    send_notification(..., urgent=True)
    sm.shelly_critical_alerted = True
    sm.transition(LOCKOUT, ...)

The flag is set on the next line, unconditionally, whether or not anything was sent. And the variable is named alerted.

It does not mean alerted. It means attempted. On March 9 and June 23 the system detected the condition, failed to reach anyone on either channel, recorded that it had reached someone, moved the pump into lockout, and never raised the subject again for the rest of the episode. The guard that exists to stop the alert repeating every thirty seconds also stopped it from ever being retried.

The only surviving evidence that anything was missed is an ERROR line in a log file — sitting on the machine that was cut off from the world.

The Fix Is a Return Statement

This one is genuinely small, which is part of why it is worth writing down. Have the send function return True when a send succeeds and False when it throws. Then gate the flag on it:

if send_notification(..., urgent=True):
    sm.shelly_critical_alerted = True

That is the whole change. Now a failed send leaves the guard open, and the next poll thirty seconds later tries again. The alert goes out whenever the network comes back — late, but sent, and with a timestamp that makes the gap visible instead of invisible. On March 9 and June 23 that single line would have converted two permanently lost emergency alerts into two late ones.

Late and honest beats silent. It is not close.

What This Means If You Are Buying Monitoring

There is a hierarchy of questions about any monitoring system, and most buyers stop two rungs early.

1. Does it detect the condition? Everyone asks this. It is where the sales conversation lives.
2. Does it route the condition to a tier that matches the severity? Fewer ask. My system routes a stuck float, its most likely real failure, to a quiet archive mailbox.
3. Does that tier deliver at all? Almost nobody asks. A blank line in a config file muted 2,550 of my notifications for eighteen days before I noticed.
4. Does that tier deliver during the specific fault it exists to report? I have never heard this asked outside of datacenter operations, and it is the one that got me twice.

Question four is different in kind. The first three you can answer on a calm afternoon by reading code and config. The fourth requires you to notice that your alert path and your fault share components, which means enumerating dependencies and looking for the intersection. It is not hard. It is just not a thing anyone thinks to do, because the alert path works perfectly every time you test it on a day when nothing is wrong.

Which is the trap. You cannot test an alert path by sending a test alert. A test alert on a healthy network proves the healthy-network case, and the healthy-network case was never in doubt. The only test that means anything is to break the thing and see if you hear about it.

So: unplug your internet. Wait for whatever is watching your building to notice it has gone blind. See whether anything reaches your phone.

I have never run that test on my own system. That is exactly why a March outage and a June outage both went unnoticed for six months and three months respectively, until I went looking through a log file this week for a different reason entirely.

I am publishing the answers for my own system while they are still embarrassing, because a monitoring product whose failure modes are undocumented is not a product. It is a hope with a subscription attached.

My honest answers today: the alert travels over the same uplink as everything else, and no, it does not try again. The second one gets fixed with a return statement. The first one needs an alert path that does not depend on the building's internet, which at this scale is a $10 to $15 a month cellular link — not a rounding error, but not an enterprise line item either.

The plug in my basement has now been running the pump on a timer for six days straight, 682 cycles, because a ten dollar float switch is stuck. The system is handling it. If the network drops while it is handling it, I have two channels to tell me, and today I learned that is a longer way of saying I have none.