My Heartbeat Said Everything Was Fine. Nothing Had Been Delivered in Two Days.

This is a correction of yesterday's post. In it I described my sump monitor going quiet for twenty-two hours, explained why, and prescribed a fix. Today I found the artifact that proves the fix would have done nothing, found fifteen more dropped messages I had missed, and had to walk back a theory I had confidently retracted. All three corrections point at the same mistake, so it seemed worth writing down rather than quietly editing.

The float switch in my sump pit stuck at the end of July. Since then my software has run the pump on a timer, two minutes on and ten minutes off, and emailed me about every cycle. Two days ago it sent number 3,509 and then stopped.

Yesterday I wrote that up. My conclusion was that I had built the right checks and left them inert, and my top recommendation was to wire the monitor's heartbeat age to an actual alert:

Add one branch — if hb_age is not None and hb_age >= 900: alert(state, "monitor_down", ...) — next to the five alert calls already in that file.

That is a reasonable-sounding fix. It is also useless against the thing that actually happened, and I can prove it with a message my own system sent me while I was asleep.

The Log Line That Kills the Fix

My daily digest is a separate program from the monitor. It went out at 12:00:03 UTC on August 21, which is twenty-two hours and forty-five minutes into the blackout. Here is what it thought of the situation:

OVERALL: OK

  monitor     : state=TIER_3  cycles=3621  last_run=0.2h ago

PRIMARY SERVICES
  monitor svc : active   (heartbeat 28s ago)
  guardian    : active
  smart-assess: active
  last guard  : ... monitor_heartbeat_age_s=10 ...

Ten seconds.

In the middle of a two-day communications blackout, the heartbeat I proposed alarming on was ten seconds old. The branch I wanted to add would have evaluated 10 >= 900, concluded everything was fine, and gone back to sleep. It would have done that roughly every twelve seconds, for two days, and it would have been right every single time.

The monitor was never down. It was running the whole time, holding the pump's schedule to the second, writing its heartbeat file, advancing its cycle counter from 3,509 to 3,621. It was alive and healthy and completely unable to make itself heard, and none of the things I built to watch it were looking at that.

The Fifteen Messages I Did Not Look For

Yesterday I described a channel that "had not missed a twelve-minute beat since July 22." I got that by finding where the stream ended. It did not occur to me to check whether it had holes in it.

It did. The cycle numbers are a dense sequence, so gaps are trivially checkable, and pulling every message sent on August 20 turns up this:

Fifteen messages were already gone before the one I noticed. The channel did not stop; it degraded for eleven hours, recovered for about two, and then stopped for good.

That shape matters. Intermittent single drops, then a longer outage, then a brief recovery, then permanent failure — that is what running into a rate limit looks like from the inside. It is not what a crashed process looks like. A crashed process stops once.

The Theory I Retracted, and Why the Retraction Was Wrong

Here is the part I am least comfortable with. Yesterday I considered a send-quota explanation, and dismissed it in print:

The quota event and the silence were three hours apart and unrelated. During the twenty-two-hour silence, that same account sent at least a dozen other automated messages. Every other robot in the house was talking normally.

The other robots were indeed talking. They also could not possibly test the hypothesis, because they do not use the same send path. I checked this morning, which took about ten seconds:

$ grep -rl "smtplib" ~/lobster --include="*.py"
(no matches)

Every automation in my main repository sends mail through a hosted API. All three sump programs open a direct SMTP connection and log in with an app password. Those are two different transports with two different allowances. I used a dozen processes on the healthy path as a control group for a limit on the other one, and concluded there was no limit.

That is not a subtle error. It is the oldest one there is: I picked a control that could not produce a negative result, and it gave me the tidier story.

What Actually Bounced

There were four rejection notices that morning, all of them reading "You have reached a limit for sending mail. Your message was not sent." Yesterday I filed them as unrelated noise. I did not read them closely enough to notice what they were attached to.

None of the four was a sump alert. Two were notifications from an unrelated messaging bridge. The other two were personal emails I had written to a family member.

My sump pump had eaten the household's mail allowance, and the first visible casualties were my own correspondence. Then it starved itself. The first missing cycle at 09:12 comes fourteen minutes before the earliest bounce I can see, which is about what you would expect if the pump hit the ceiling first and everything else piled into it afterward.

The Asymmetry That Settles It

Three separate programs on that machine send mail through the same account, with the same credentials, using the same library. Only one of them is broken:

The two programs that want a handful of messages a day still get through. The one that wants a hundred and twenty has not landed a message in forty-six hours. The account itself is fine — I have sent normal email from it to several people over the same period.

It is not blocked. It is full, and the monitor retries every twelve minutes, taking each freed slot the instant it appears and never accumulating any headroom.

Why Nothing Told Me Any of This

The send failures are caught. Here is what happens to them:

except Exception as e:
    log(f"ERROR: Failed to send email: {e}")

The error is formatted into a string and written to a log file, on the machine whose sending is broken, reachable only through the channel that is broken. There is no counter, no state change, no escalation, and nothing that survives to the next pass of the loop. Two hundred and twenty-seven failed messages produced two hundred and twenty-seven log lines and zero signals.

Worse, there is no queue and no retry. The message is built from live state at send time. When the send fails, that reading is simply gone. Two hundred and twenty-seven cycles have now happened for which no record exists anywhere — not in an inbox, not in a database, not in a file. The only artifact that survived the entire blackout is one daily digest that says OVERALL: OK.

The Machine, Meanwhile, Is Flawless

I polled the plug every twenty seconds this morning while writing this, because after two days of being wrong about my own software I wanted to see the hardware with my own eyes.

Two minutes and three seconds of run time against a hundred and twenty second specification, which is inside my poll resolution. Steady draw of 501.8 watts with a standard deviation of 1.54. That is the third time I have measured this pump's unloaded state by a different method, after 505.3 watts from a database and 510.2 watts from a faster poll, and it is 31.5% below the 732.7 watts this pump draws when it is actually moving water.

The control loop has been metronomic for thirty-one days. A fifteen-dollar smart plug has now outlived two distinct software failures without a single reboot. Everything that has gone wrong in this incident has gone wrong in the layer I wrote.

What I Am Actually Going to Build

Yesterday's recommendation was wrong, so here is the replacement.

1. Put the check on the receiving end

A job on any machine other than the monitor host that looks at the destination inbox, finds the newest message from the pump, and alerts if it is older than thirty minutes. That is roughly five lines. It would have fired on August 20 at 13:45.

The reason it works is that it tests the property I actually care about — did the message arrive — rather than a proxy for it. It does not care whether the cause is a rate limit, a crashed process, a dead network, a bad certificate, or something I have not thought of. Any check that runs on the sending machine can only ever tell you the sending machine believes it is fine.

2. Count failures somewhere that survives the failure

One integer, consecutive_send_failures, incremented in that except block, stored in the state file the monitor already writes, and printed in the daily digest. The digest is on a working path. It has a slot for this. It has been saying OVERALL: OK for two days.

3. Stop the routine traffic from eating the emergency budget

Emailing every twelve minutes about a condition that has not changed in thirty-one days is not monitoring, it is a stuck key. Send the first three, then one an hour, and keep the full record in a local append-only file. This is the change that stops the system from taking down its own alert channel.

4. Give the urgent path its own transport

My code carefully sorts every message into URGENT or ROUTINE, and then hands both to the same account with the same single allowance. The sorting is decorative unless the paths are genuinely separate. A flood alert should not be able to queue behind a hundred and twenty status updates.

What I Would Tell a Customer

If you are paying someone to monitor a pump, a boiler, a walk-in cooler, or a building, there are four questions worth asking, and I now know all four from the wrong side.

• What happens when your system stops hearing from my equipment? Not when it detects a fault — when it detects nothing at all. Silence should be the loudest signal in the system, and it is usually the only one with no handler.
• Where does that check run? If the answer is "on the device" or "on the gateway," it cannot detect the device being unable to reach you. It has to run somewhere else.
• What is your alert path, and what else uses it? If routine status traffic and emergency alerts share a queue, a credential, or a rate limit, then a chatty sensor can silence a real alarm. Mine did.
• Where does the data live besides the alert? If the reading exists only inside the notification, then a failed notification destroys the measurement. I lost two hundred and twenty-seven of them and I sell this.

I have now been wrong about this incident in three distinct ways in four days: a threshold I never measured, a watchdog reading a signal that could not go bad, and a control group that could not falsify anything. Not one of those is an AI problem. There is no model you can put on top of a system that cannot tell you whether it is being heard.

The cheapest thing on my list is still the flashlight. I am going to go look at the pit.