My Pump's Self-Repair Routine Has Run 747 Times. It Last Worked in April.

Yesterday I wrote about discovering that my sump pump monitor had been performing active mechanical repairs on my equipment and filing every record of them in a channel nobody was reading. I ended that post with a prediction. This is what happened to it.

Here is how I closed:

The unstick routine will run again around eight this morning. It will be attempt twenty-four. Based on the previous twenty-three, it will fail, and it will be the first one that anybody hears about.

Half of that was right. It ran at 08:30. It failed. So did attempts 25, 26, 27, 28 and 29 over the following twenty-four hours. Nobody heard about a single one of them, because I did not go fix the configuration file, and it still sits there at 1,183 bytes with a modification time of Aug 20 09:16, five days stale, exactly as it did when I wrote the sentence promising otherwise.

But that is not what this post is about, because this morning I finally asked the question I had somehow never asked in five months of running this thing.

The Question I Never Asked

Some background, briefly. The float switch in my sump pit stuck at the end of July. Since then my software has been running the pump on a timer instead: two minutes on, ten minutes off, forever, 4,092 times so far. It is a workaround, not a fix, and it works.

Alongside the timer there is a repair routine. Every four hours the monitor stops the duty cycle and tries to shake the float loose mechanically: power off for ten seconds to let the water settle and the float drop under gravity, then six rapid on-five-seconds, off-five-seconds cycles to jolt the motor and churn the pit, then sixty seconds of holding it on to see whether the pump shuts itself off. If it does, the float is free and we are back to normal operation. If it does not, we go back to the timer and try again in four hours.

I have been writing about the fact that these attempts were being silenced. What I had not done was ask whether they were working.

$ grep -c "UNSTICK ATTEMPT"   sump_pump_monitor.log
747
$ grep -c "UNSTICK FAILED"    sump_pump_monitor.log
595
$ grep -c "UNSTICK: aborting" sump_pump_monitor.log
148
$ grep -c "UNSTICK SUCCESS"   sump_pump_monitor.log
4

And the four are not distributed the way you would hope. They are:

[2026-03-22 15:50:56] UNSTICK SUCCESS: pump idle (0.0W) after sequence!
[2026-04-30 00:58:30] UNSTICK SUCCESS: pump idle (0.0W) after sequence!
[2026-04-30 04:48:42] UNSTICK SUCCESS: pump idle (0.0W) after sequence!
[2026-04-30 14:04:10] UNSTICK SUCCESS: pump idle (0.0W) after sequence!

One in March. Three inside a thirteen-hour window on April 30. Nothing since.

Since two in the afternoon on the thirtieth of April — one hundred and seventeen days ago — that routine has run 519 more times and succeeded exactly zero times.

I want to be fair to the code here, because the four look real. The one at 00:58 on April 30 transitioned the state machine into a cooldown period and then logged COOLDOWN: Pump idle for 90s — confirmed unstuck! ninety seconds later. That is a genuine recovery, confirmed by a second observation. The routine has worked. It just has not worked since spring, and three of its four lifetime wins landed on the same day, which is what a float freeing itself on its own looks like at least as much as it looks like a repair succeeding.

What It Costs to Fail 743 Times

The reason this matters more than a bad statistic is that the sequence is not free, and I had been treating it as though it were.

Every attempt is seven motor starts. Six from the rapid cycles, one from the confirmation hold. And a motor start is not a neutral event. It is the single highest-wear, highest-inrush moment in the life of an electric pump. Continuous running is comparatively gentle; it is the starts that kill motors. Ask anyone who specifies duty cycles for a living and they will tell you the number of starts per hour is the constraint they design around.

$ grep -c "UNSTICK: rapid cycle"  sump_pump_monitor.log
4038
$ grep -c "holding ON for 60s"   sump_pump_monitor.log
599
$ grep -c "Plug turned ON"       sump_pump_monitor.log
18380

Four thousand and thirty-eight, plus five hundred and ninety-nine, is 4,637.

The ratio is not a historical artifact, either. Since the notification channel went quiet five days ago: 781 motor starts, 195 of them from unstick attempts. Same quarter. It is still running right now, at the same rate, for the same return.

Why It Cannot Possibly Know

Here is the loop that schedules it. This is the whole decision:

since_unstick = time.monotonic() - sm.last_unstick_at
if since_unstick >= UNSTICK_INTERVAL_SECONDS:
    if attempt_float_unstick(sm, "periodic TIER_3"):
        return  # unstick worked

The function returns True on success and False on failure. On True, the caller returns early and moves into cooldown. On False — which is what happened 595 times — the return value is simply discarded. Nothing catches it. Nothing counts it.

The only state the routine writes down about itself is one line:

sm.last_unstick_at = time.monotonic()

A timestamp. Not an outcome. So I went looking for the counter I assumed must exist somewhere:

$ grep -n "unstick_fail\|unstick_count\|unstick_attempts" sump_pump_monitor.py
$

Nothing. There is no failure counter, no consecutive-failure threshold, no backoff, no escalation, no give-up. And the galling part is that the pattern exists elsewhere in the same file — the state machine tracks a consecutive_dry streak to decide when to exit a tier, with a threshold and everything. I knew how to write this. I wrote it two hundred lines away. I just never applied it to the repair path.

So the system knows precisely how long it has been since it last tried something. It has never once known whether trying works.

That distinction is the whole post. A retry loop with no memory of its own outcomes is not resilience. It is a habit. It will run at a 50% success rate and a 0% success rate with exactly the same enthusiasm, forever, and it will describe both of them to you in identical language.

The Channel That Works Has Never Mentioned It

There is one more layer to this, and it is the part that has changed how I think about dashboards.

Through this entire month of failures, one thing has never broken: the daily digest. It runs on its own timer, with its own hardcoded recipient list, completely independent of the configuration file that got muted — which is exactly why it survived. It has arrived at 08:00 every single morning. Here is yesterday's, in full:

Sump system daily status — 2026-08-24 08:00:01 (from toddllm, primary)

OVERALL: OK

PUMP / SHELLY
  relay       : OFF   power 0W   temp 47C   123V
  monitor     : state=TIER_3  cycles=3978  last_run=0.1h ago

PRIMARY SERVICES
  monitor svc : active   (heartbeat 28s ago)
  guardian    : active
  smart-assess: active
  last assess : HIGH_INFLOW INFO | Pumping a lot but pump is healthy (8 min/hr)

Then I grepped the entire digest archive:

$ grep -ci "unstick" sump_daily_status.log
0

Zero. Ninety-six kilobytes of daily reports, going back months, and the one channel in this system that has never failed has never once mentioned the repair routine.

And notice that it is not lying. Every number in that digest is accurate. The relay really is off, the power really is zero, the temperature really is 47 °C, the voltage really is 123. There is even a small AI assessor in there that looks at the readings and concludes the pump is healthy, and it is correct — the pump is healthy. The digest answers the question "what is this equipment doing right now," and the answer is genuinely, boringly fine.

The problem is that nothing anywhere in my stack answers the question "what has this system been trying to do, and has any of it worked." That is not a reading. It is a rate, and a rate requires history, and nothing here keeps any.

Time to detect the muted channel: three days. Time to detect the 0.54%: five months.

The Failure Mode Nobody Builds For

I have now been publicly debugging this system for a month, and I have kept a table of what went wrong and how each one surfaced. Today's entry is different from the others in a way worth spelling out.

Six problems. Zero exceptions raised. But the first five were all variations on a message that failed to arrive. This last one is not.

These messages arrived. For the four months before anything got muted, every single UNSTICK FAILED notice was delivered to my inbox precisely as designed. I received hundreds of them. I read a good number of them. And I noticed nothing at all, because a failure that shows up alone, once every four hours, phrased as "attempt failed, will retry", is indistinguishable from weather. It reads as the system working. In a sense it is the system working — that is the designed behavior on the failure path.

The information was never missing. It was never aggregated. Nobody ever divided one number by another.

What I'm Changing

Every retry loop gets a success rate and a stopping condition. Two counters, not a timestamp. last_unstick_at answers "when did I last try," which is the wrong question asked well. attempts and successes answer "does this work." Any loop that cannot answer the second question will run forever at any success rate, including zero, and will sound the same the whole time.

Worth being precise about what "stopping condition" means here, because I am not going to stop the attempts. A one-in-187 shot at freeing that float is still worth seven motor starts every four hours, given that the alternative is running the pump on a timer indefinitely. What has to change is not the action, it is the sentence. After a dozen consecutive failures, the routine should stop saying "attempt failed, will retry" and start saying "this repair has not worked in 117 days, and something else is wrong." Right now attempt 748 will describe itself in exactly the same words attempt 3 used back in March.

A dashboard that reports only instantaneous state will pass every slow failure it ever meets. Mine has said OVERALL: OK every morning of this incident and has been telling the truth each time, because at 08:00:01 every reading is in range. The thing that is wrong is not a value, it is a trend, and it will never appear on a display that only knows about now. If you are evaluating a monitoring product, ask to see a rate and a history next to every current value. If all it can show you is now, it can only catch failures that happen all at once.

Cheap actions get budgeted like free ones. The unstick sequence costs nothing in dollars and about two and a half minutes in time, so it got scheduled every four hours in March and never re-examined. Priced honestly it costs seven motor starts, and it has now spent a quarter of my pump's lifetime start budget chasing a one-in-187 shot. So: log the cost of every automated action in the units the equipment actually cares about — starts, cycles, thermal excursions, hours — and never in wall-clock time. Anything whose true cost is invisible will be over-used, and by something that never gets tired.

One Question for Anyone Buying This

If you are looking at condition monitoring for a building or a piece of equipment, this whole month collapses into a single question worth asking every vendor who pitches you:

If they can produce it, they are measuring outcomes and somebody there has thought about this. If they can only tell you how often the action fires — how many alerts, how many interventions, how many optimizations applied — then they are measuring activity, and you will discover the difference eventually, the way I did, with a grep -c five months too late.

Meanwhile, in the Basement

The pump ran again at 06:54 this morning. Cycle 4,092, drawing 489.7 watts at 48.0 °C on 122.4 volts, with the smart plug reporting 827 hours of uptime and zero reboots since the day it was installed.

A fifteen-dollar plug and about a hundred lines of control logic have now run flawlessly through six separate failures of the monitoring software I wrapped around them. Every one of those failures was in my layer. Not one of them was an AI problem, and not one of them would have been prevented by a better model. You cannot fix a system that is not counting by making it smarter.

There is one genuinely encouraging thing in the data, and I am going to flag it rather than claim it. Daily average power has drifted up over the month, from 484 watts to 494, while daily average temperature has drifted down from 50.4 °C to 48.4, and thermal warnings have collapsed from 233 in a single day two weeks ago to two today. More power at a lower temperature is what a submersible pump moving more water looks like, since the water is what cools the motor. That is probably rain. But I cannot actually tell you, because I still have no structured log of readings to join against a weather record — I produced that trend this morning by running a regular expression over free text, which is a large part of why it took thirty-four days to look.

The routine will run again around 08:30. It will be attempt 748. Based on the previous 519, it will fail, and it will report that failure in the same untroubled language it has used since March, to a channel with nobody on the other end.

And the cheapest instrument on my list is still a flashlight. Thirty-four days in, it retains a perfect record, and it is still the only tool in this entire story that I have not used.