My System Tried to Fix Itself 23 Times. It Told No One.

This is a correction to yesterday's post, which was itself a correction to two posts before it. Yesterday I found the line of configuration that had silenced my monitor for three days, and then I defended it. I wrote that the mute was surgical, that it had hit only routine noise, and that the important channel was untouched. I had counted the suppressed messages. I had not read them.

The float switch in my sump pit stuck at the end of July. Since then my software has run the pump on a timer, two minutes on and ten minutes off, and it has now done that 3,974 times. On the morning of August 20 someone with access to that machine added one line to a configuration file, restarted the service, and turned off the channel that had been narrating all of it.

Yesterday I worked out that this had happened and wrote about it. I ended that post satisfied. The intervention had been aimed at a firehose of cycle notifications, it had hit the firehose, and the urgent path had survived. Sloppy, undocumented, but essentially correct.

This morning I ran a different command. Instead of asking how many messages were discarded, I asked which ones.

What Was Actually in the Bucket

442  Plug turned OFF                              (routine cycle)
 67  TEMP WARNING: 50.XC approaching limit (60.0C)
 23  UNSTICK ATTEMPT: starting rapid cycle sequence
 21  UNSTICK FAILED: pump still running (49X.XW)

The first line is what I expected and what I defended. The last two are not status updates at all.

An unstick attempt is my software performing a physical repair on my equipment. Every four hours, it stops timer-driven pumping and runs an intervention designed to jog the stuck float loose: power off for ten seconds, then six rapid on-five-off-five cycles, then a sixty-second hold to test whether the float has cleared.

[04:16:56] UNSTICK: rapid cycle 3/6 complete
[04:17:06] UNSTICK: rapid cycle 4/6 complete
[04:17:16] UNSTICK: rapid cycle 5/6 complete
[04:17:27] UNSTICK: rapid cycle 6/6 complete
[04:17:27] UNSTICK: holding ON for 60s to check if float cleared...
[04:18:27] UNSTICK FAILED: pump still running (491.2W)

That ran at four o'clock this morning. It is the twenty-third time. It is the twenty-first failure. There have been zero successes.

Twice, the sequence did not even finish. It aborted halfway through because the motor had gotten too hot to keep going:

[11:28:13] UNSTICK: aborting — temp 50.7C too high
[15:28:27] UNSTICK: aborting — temp 50.3C too high

My system started a mechanical repair on a pump, got three cycles in, measured the motor, decided the equipment was too hot to continue, stopped, and filed that decision somewhere nobody was looking. It made a good call. It made it alone.

Nobody Decided This

The part that has stayed with me all morning is that there is no bad decision anywhere in this story. There is no line of code where someone classified a failed mechanical repair as unimportant.

Here is the call that sends the unstick notification:

log(f"UNSTICK ATTEMPT: starting rapid cycle sequence ({reason})")
send_notification(
    f"Sump pump: attempting float unstick ({reason})",
    f"Sequence: OFF 10s, then {UNSTICK_CYCLES}x (ON 5s / OFF 5s), ...",
)

And here is the function it calls:

def send_notification(subject, body, urgent=False):
    ...
    recipients = NOTIFY_EMAILS_URGENT if urgent else NOTIFY_EMAILS_LOG

That is the whole mechanism. The call site does not pass a severity, so it takes the default, and the default is the quiet channel. The same is true of the UNSTICK FAILED call fifty lines later.

I wrote that default months ago, when the only things this function sent were cycle notifications and routine status. It was correct then. It was a reasonable, even tasteful, piece of API design: make the common case terse, let the exceptional case opt in. Then I added self-repair to the system, wrote two perfectly ordinary-looking function calls, and inherited a decision from a keyword argument I was not thinking about.

A default argument is a decision you make once, in a function signature, that then applies silently to every call site anyone writes afterward — including the ones written months later, by someone who never read the signature. Even when that someone is you.

So when a person came along in the middle of an incident and muted the routine channel, they were not making a mistake either. They were muting what they believed the channel contained. The bucket had quietly acquired new contents in the interim, and nothing anywhere announced that.

The Number That Didn't Move

There is a second thing I have to report, and it is less comfortable than the first.

Yesterday I published the root cause. I gave the exact file, the exact line, and a two-line fix. I ended with seven action items and put the config change first. That was twenty-four hours ago.

$ grep -c "LOG email skipped" sump_pump_monitor.log
553

$ ls -la ~/home-assistant/.env
-rw------- 1 tdeshane tdeshane 1183 Aug 20 09:16 .env

Yesterday the count was 423. The file's modification time has not changed since the morning it broke. One hundred and thirty more notifications were discarded in the twenty-four hours after I published the fix for why they were being discarded.

The daily rate is what makes the point. 134, then 127, then 124, then 129. Finding the root cause moved that line by nothing, because a diagnosis is not a control action. The system does not read blog posts.

I mention this because I got a newsletter this week that put a number on the same phenomenon at a scale where it costs real money. Nexus Labs interviewed an energy manager at Texas Tech who, in his program's first year, identified $977,000 in energy savings across his buildings and had the capacity and permission to actually capture $97,000 of it.

Ten percent. At an institution with a funded energy program and a person whose actual job it was.

That is the same failure I am describing, scaled up by five orders of magnitude. The monitoring worked. The analysis worked. What was missing was throughput between knowing and doing — and every dollar spent on better detection landed on the wrong side of that gap. It is worth being honest that the monitoring industry, mine included, sells detection and bills for detection, while the customer's actual constraint is usually the authority and the hours to act on what was detected.

Five Failures, Zero Errors

This is the fifth distinct way my monitoring has failed in thirty-three days. Here they all are:

Five failures. Zero exceptions. Not one of these would have been caught by better error handling, because not one of them was an error. Every single one was the system doing precisely what it was written to do.

That is the thing I would most want a small business owner to understand before buying condition monitoring from anybody, me included. The dangerous failures in these systems are not crashes. Crashes are loud, and everyone already builds for them. The dangerous ones are the paths where nothing goes wrong: the else branch that fires four hundred times and increments no counter, the default parameter, the threshold nobody measured, the channel that was correct to mute in March and wrong to mute in August.

What I'm Changing

Three things, in the order I would do them again.

Severity becomes a required argument. Not a default. If send_notification demands that every call site state how important the message is, then adding self-repair to the system would have forced somebody to answer the question out loud. Where a default is unavoidable, it should be the loud one, so that forgetting is noisy instead of silent. Omission should never be the quiet option.

The dashboard reports on the automation, not just the equipment. My daily digest arrived on schedule every morning of this incident saying OVERALL: OK, and it will say it again at eight o'clock today with 553 messages in the bin behind it. It is not lying. It is reporting on the pump, and the pump is genuinely fine. Nobody ever asked it to have an opinion about the software. The change on my list is about ten lines against a path that already works: print, on every run, which channels have recipients configured, how long any channel has been empty, and how many messages that emptiness has eaten.

Count a channel before you mute it. Thirty seconds with grep -c before the restart would have shown 44 self-repair records sitting in the bucket I was about to empty. Every real system has a mute, mutes get used during incidents, and an incident is the exact moment when nobody has thirty seconds. So the check has to be built in: a mute that reports what it is about to silence, and expires on its own.

Meanwhile, in the Basement

The pump ran again at 07:04 this morning. Cycle 3,974, drawing 492 watts at 48.4 °C, on 123.5 volts, with the plug reporting 803 hours of uptime and zero reboots since it was installed.

A fifteen-dollar smart plug and about a hundred lines of control logic have now run flawlessly through five separate failures of the monitoring software I wrapped around them. Every failure was in my layer. Not one of them was an AI problem, and not one would have been prevented by a better model. There is nothing you can add to a system that has stopped talking and does not know it.

The unstick routine will run again around eight this morning. It will be attempt twenty-four. Based on the previous twenty-three, it will fail, and it will be the first one that anybody hears about.

And the cheapest instrument on my list is still a flashlight. Thirty-three days in, it has a perfect record, and it is still the only tool here I have not used.