Yesterday I published a post arguing that my sump pump monitor had been welded into its maximum emergency state for forty-two days, that both of its designed exits required the same physical event, and that the only things which had ever reset it were a float release in June and two occasions when the smart plug fell off the network. My conclusion was that the recovery path depended on hardware failure, and that a state with no time-based exit is not a state but a destination.
Four hours after I posted it, the recovery path fired on its own and worked.
Then the system climbed all the way back to maximum in two hours and thirteen minutes.
I have spent this morning going through what happened, and it turns out the interesting part is not that the fix worked. It is that "the fix worked" is a sentence my monitoring is fully capable of producing and completely unable to qualify.
What Happened
Some background, briefly, for anyone arriving here first. My basement sump pump runs on a $15 smart plug with a power meter in it. A monitor watches wattage. If the pump runs continuously past a threshold, that usually means the float switch is stuck closed, so software steps through a protective ladder: power cycle, then duty-cycle the plug on progressively harder schedules, with a top tier of 120 seconds on and 10 minutes off. Every four hours while it is up there, it also attempts an "unstick" — six rapid on/off cycles meant to jolt the float loose mechanically.
It had been at the top tier since July 21st. The unstick had failed 252 consecutive times.
Here is attempt 253.
[2026-09-02 10:52:49] TIER_3: periodic unstick attempt (last was 4.1h ago)
[2026-09-02 10:52:49] UNSTICK ATTEMPT: starting rapid cycle sequence
... six rapid ON/OFF cycles ...
[2026-09-02 10:54:47] UNSTICK: holding ON for 60s to check if float cleared...
[2026-09-02 10:55:49] UNSTICK SUCCESS: pump idle (0.0W) after sequence!
[2026-09-02 10:55:49] STATE: TIER_3 -> COOLDOWN (unstick attempt succeeded)
[2026-09-02 10:57:20] COOLDOWN: Pump idle for 91s — confirmed unstuck!
[2026-09-02 10:57:20] STATE: COOLDOWN -> NORMAL (cooldown confirmed)
Nobody touched it. There is a manual trigger file for exactly this, and grepping the entire log for the word manual returns zero hits across six months. This was the scheduled four-hourly attempt, running unattended, and for the first time since April it did what it was written to do.
Every statement in that log excerpt is true. The pump really was drawing 0.0 watts. It really was idle for 91 seconds. The float really had released.
And then:
[2026-09-02 11:04:22] STATE: NORMAL -> POWER_CYCLE (pump ran 6.0 min, wet threshold 6.0 min) [2026-09-02 11:05:32] STATE: POWER_CYCLE -> TIER_1 (power cycle failed) [2026-09-02 11:57:49] STATE: TIER_1 -> TIER_2 (escalation from TIER_1) [2026-09-02 13:09:13] STATE: TIER_2 -> TIER_3 (escalation from TIER_2)
| The recovery, measured | |
|---|---|
| Time in NORMAL | 7 minutes, 2 seconds |
| NORMAL to back at maximum tier | 2 hours, 13 minutes |
| Confirmation window that certified the fix | 91 seconds |
| Duty cycles completed since re-entry | 88 |
| Of those, cycles where the pump drew power | 88 — 100% |
| Cycles under the 100 W "dry" exit threshold | 0 |
| Lowest wattage seen in any pulse | 465.9 W |
| Urgent notifications sent about any of this | 0 |
The Success Rate I Had Never Computed
Once the routine had actually succeeded, I got curious about how often it does. I had been describing it to myself, for six months, as the recovery mechanism.
| Unstick routine, lifetime | |
|---|---|
| Sequences started | 802 |
| Logged as succeeded | 5 |
| Logged as failed | 322 |
| Success rate | 0.62% |
Five successes in six months: one in March, three on a single day in April, and this one. Four months separated the fourth from the fifth.
I would never knowingly ship a recovery routine that works six times in a thousand. I shipped this one and ran it for half a year because I never measured it, and I never measured it because it announces its successes and swallows its failures. UNSTICK SUCCESS is a capitalized line with an exclamation point. The failures are a quiet log-tier email that, for the last two weeks, has not even been going anywhere.
This is the shape of the problem generally. Recovery routines are written by optimists and read by nobody. The success path gets the loud log line. The failure path gets a counter that no dashboard plots.
The timing here is almost unfair. This same week, Anthropic previewed something called the Model Hardware Standard, a specification for letting AI agents discover and operate physical instruments through standardized read and write primitives. Buried in the results is a number from QuEra Computing: an automated laser recovery routine that went from a 58% success rate to 99.3%.
Someone measured their recovery routine, found it at 58%, and considered that a problem worth solving. Mine has been running at 0.62% and I found out by accident.
The Question I Should Have Been Asking
Knowing the routine is unreliable is useful. What is more useful is the question the seven minutes forced: when recovery does work, how long does it last?
I had never asked. So I pulled every entry into the NORMAL state across six months and measured how long each one held.
| NORMAL residency, 74 completed periods | |
|---|---|
| Median | 38.6 minutes |
| Mean | 4.29 hours |
| Shortest | 4.0 minutes |
| Longest | 92.4 hours |
| Under one hour | 46 of 74 (62%) |
| Over 24 hours | 4 of 74 (5%) |
The median recovery on this system lasts thirty-nine minutes.
Now look back at the log excerpt. The cooldown state confirmed the fix after the pump stayed idle for 91 seconds. Ninety-one seconds is one twenty-fifth of the median time before the same condition comes back.
If your confirmation window is shorter than the median return time of the fault, you are not confirming a fix. You are sampling a gap.
That is the entire defect, and it is not specific to sump pumps. It is in every retry-until-it-looks-fine loop I have ever written, and in most of the ones I have read.
The Ladder Is a Stopwatch
There is one more thing in the data, and it is the part that made me stop defending the design.
Twenty-nine times since February, the system has entered NORMAL and then climbed all the way back to maximum tier. Here is how long that took each time, most recent last:
| Entered NORMAL | Reached maximum tier after |
|---|---|
| 2026-04-19 | 2.50 h |
| 2026-04-30 | 2.58 h |
| 2026-04-30 | 2.81 h |
| 2026-05-02 | 2.19 h |
| 2026-06-15 | 2.39 h |
| 2026-06-23 | 2.21 h |
| 2026-07-21 | 3.53 h |
| 2026-09-02 | 2.20 h |
That is not a distribution produced by a pump. It is a distribution produced by a timer. Here is the escalation config, verbatim from my own source:
TIER_1: 60s ON / 900s OFF, escalate_after 3 cycles -> ~48 min TIER_2: 90s ON / 600s OFF, escalate_after 6 cycles -> ~69 min
Add the six-minute wet threshold and the seventy-second power cycle and you get roughly 2.1 hours from NORMAL to maximum, by construction. Yesterday tier one held 52 minutes and tier two held 71 minutes, right on the design.
The escalation counters count cycles. They do not count evidence. Nothing between NORMAL and maximum duty asks a new question about the pump, and the pump answers the same way every time anyway: wet, every cycle, never once under 465 watts.
If every path through your escalation ladder takes the same wall-clock time regardless of what the equipment is doing, the ladder is a delay. You could replace it with a sleep statement and not change a single outcome.
I want to be fair to the design, because there is a version of this that overcorrects. Gradual escalation is genuinely better than jumping straight to maximum duty; it gives a marginal float several chances to release under progressively gentler conditions, and on a pump that is only intermittently stuck it would work. My equipment is not intermittently stuck. Six months of data say so, and the design has no way to notice that the ladder has stopped being a diagnosis and started being a countdown.
Three Things Worth Running Against Your Own Logs
None of this needs my hardware. If you have anything with a log and a mode, these are each about ten minutes.
- Compute the success rate of every automated recovery routine you own. Not whether it works — the fraction. Count the starts, count the successes, divide. If you have never done this for a routine that has been running for months, you do not know what it does. You know what it was supposed to do.
- Measure dwell time in the healthy state, not transitions into it. Bucket every entry into your normal state and how long it held, then look at the median. A system whose median healthy period is thirty-nine minutes is not recovering, it is oscillating — and every dashboard plotting "recoveries this month" will show that number rising as the system gets worse.
- Check whether your escalation measures anything. Time each path through your ladder across every episode you have. If the durations cluster tightly, they are coming from your config, not from your equipment.
What I'd Tell a Building Owner
Every maintenance system I have ever worked with can tell you that a problem was resolved. Almost none of them can tell you whether it stayed resolved.
Your building management system logged the alarm clearing. It did not log that the same alarm returned forty minutes later, and then again, seventy-four times, because each return is a fresh event and events do not remember each other. The work order was closed. The condition was not.
This is why I keep every raw sample off every device I deploy, and it is why the number I care most about on a customer site is not detection latency or alert volume. It is time to recurrence: when this fault clears, how long before it comes back, and is that distribution getting worse?
For a stuck float in my basement, the answer is thirty-nine minutes, and it took six months of retention and one accidental success to find out. For a valve stuck in override, an air handler on a backup schedule, or a compressor that keeps clearing its own fault, the answer is worth considerably more than the sensor that detected it.
So here is the question I would put to any owner, and it is the follow-on to the one I asked yesterday: the last five equipment problems your building reported as resolved — how long did each of those resolutions last? If your monitoring cannot answer that, then "resolved" in your system means "stopped complaining," and those are not the same thing.
Mine could not answer it either. I have the data. I just spent six months plotting the wrong number.
Want monitoring that tells you whether the fix held?
We deploy edge AI monitoring for small buildings using off-the-shelf sensors — no enterprise contracts, no cloud lock-in, full retention of raw device telemetry, and time-to-recurrence reporting on every fault your equipment clears by itself.
See What We BuildMore from this series: Forty-two days at maximum, and the exit requires a crash | The panics stopped, nothing was fixed | The escalation ladder was two hours tall | The self-repair routine ran 747 times
Sources: all figures from my own deployment — a 143,502-line monitor log spanning 2026-02-23 to 2026-09-03, its 358 state-transition records, the monitor and state-machine source, and a live Shelly.GetDeviceInfo query against the plug on 2026-09-03. The QuEra laser-recovery figures (58% to 99.3%) are from Anthropic's Model Hardware Standard research preview announcement, published 2026-08-28.