Ninety Seconds to Declare Recovery. Thirty-Four Minutes Until It Came Back.

Fifty-three times since March, the monitor watching my sump pump has written this line to its log:

COOLDOWN: Pump idle for 90s — confirmed unstuck!

Then it returns to its NORMAL state and sends a message with the subject "Sump pump: float unstuck, back to normal."

I pulled every state change out of the log this morning, 388 of them across 186 days, and measured how long each of those recoveries actually held. The median was 34 minutes.

The test is ninety seconds long. The thing it is testing for comes back, on a median, twenty-three times later than that. My monitor has not been recovering fifty-three times. It has been passing a test that is too short to fail.

The Ladder

The control layer is a state machine on a Linux box, driving a $15 smart plug that measures watts. If the pump runs longer than it should, the monitor cuts power and restores it, and if that does not settle things it climbs a ladder of timed duty cycles, each one giving the pump less continuous run time and a longer rest:

That dash in the last row is the whole story, and it is two words in the source:

TIER_3: {
    "run_s": 120,
    "rest_s": 600,
    "escalate_after": None,
    "next": None,
},

TIER_3 is the top. There is nothing above it. I knew that when I wrote it, and at the time it seemed fine, because I assumed the ladder was a path to somewhere. What I had not looked at was where the ladder actually goes when it runs out of rungs.

Where the Six Months Went

Here is every state, and the share of those 186 days my system spent sitting in it.

The emergency state is the resting state. My maximum-intervention tier, the one whose comment in the source says "Max duty, alerts every cycle," has held ninety percent of six months. NORMAL is the exception, at seven percent.

I have written before about being stuck at the top of this ladder, and about needing a transition that fires on a clock. But that framing was wrong in an interesting way, and the full transition history is what corrected it. The system is not stuck. It leaves TIER_3 constantly.

Thirty-Four Minutes

I measured all 78 NORMAL episodes in the log, from the moment the monitor declared normality to the moment it left again.

Two thirds of my recoveries did not survive an hour. Eleven of them did not survive ten minutes. The shortest lasted four.

This is what a ninety-second criterion buys you. The condition I am watching for recurs on a timescale of tens of minutes, and I am sampling for it in a window a minute and a half wide. A pump that is idle for ninety seconds tells me almost nothing, because a pump with water still arriving is also idle for ninety seconds, between fills. My test does not distinguish a fixed pump from a pump between cycles. It never could.

The night of September 9th shows it cleanly. Watch the timestamps.

18:51:32  TIER_3   -> COOLDOWN   (pump stopped during ON pulse)
18:53:03  COOLDOWN -> NORMAL     (cooldown confirmed)
18:59:33  NORMAL   -> POWER_CYCLE (pump ran 6.0 min)
19:00:43  POWER_CYCLE -> TIER_1
19:53:57  TIER_1   -> TIER_2
21:05:12  TIER_2   -> TIER_3

Ninety seconds of quiet earned a "back to normal" at 18:53. Six and a half minutes later the pump had run six minutes straight and the whole climb started again, arriving back at the top of the ladder two hours later. Then it did it twice more: recovered at 00:48, climbing again by 01:48; recovered at 22:03 the next night, climbing again by 22:25. Three full ascents in thirty hours, each preceded by a logged success.

My favorite part of that night is a little earlier, just after midnight, when the cooldown check ran five times in half an hour:

00:03:53  TIER_3 -> COOLDOWN      00:03:57  back to TIER_3   (4 s)
00:14:57  TIER_3 -> COOLDOWN      00:15:10  back to TIER_3  (13 s)
00:25:52  TIER_3 -> COOLDOWN      00:25:56  back to TIER_3   (4 s)
00:36:31  TIER_3 -> COOLDOWN      00:36:40  back to TIER_3   (9 s)
00:47:07  TIER_3 -> COOLDOWN      00:48:38  NORMAL          (91 s)

Four failures and a pass, inside forty-five minutes, with nothing changed in between. The fifth attempt did not succeed because the pump was fixed. It succeeded because the water happened to take ninety-one seconds to come back instead of nine. My recovery signal is the arrival time of the next slug of groundwater, which is weather, not repair.

Every Climb Starts From Zero

The second half of the problem is that none of those fifty-nine climbs knows about the other fifty-eight. From the state machine:

if new_state in TIER_CONFIG:
    self.cycle_count = 0
    self.consecutive_dry = 0

Entering any tier zeroes the counters. There is no episode counter anywhere in the object, no history, nothing that accumulates across a recovery. So every climb re-derives the situation from scratch: three cycles at TIER_1 to decide TIER_1 is not working, six more at TIER_2 to decide the same, and then it arrives at the top rung it was on an hour ago and settles in.

That costs about two hours of low-duty pumping per climb, during which the pump is deliberately being given less run time than the situation calls for. Twenty-seven days of my six months have included at least one of those. The information that would have skipped the whole staircase, that this exact climb has already happened dozens of times, is sitting in the log file and is not available to the thing that needs it.

The Door That Would Have Called Me

There is one state in the machine that means "stop, a person is required." It is LOCKOUT. I went through the source to find every way in, and there are exactly three:

1. Plug temperature at or above 70 °C, the hard electronics cutoff.
2. Plug temperature above 60 °C, the soft cooling threshold.
3. The plug unreachable on the network for 30 minutes.

Read that list again for what is missing. There is no entry that says "the condition this ladder exists to resolve has not resolved." The pump-driven path tops out at TIER_3 and loops back to NORMAL. It cannot reach the one state that would summon me, because persistence of the original fault is not one of the three doors.

The log agrees. LOCKOUT has been entered seven times in six months, and all seven read the same way:

2026-03-09 12:07  TIER_1 -> LOCKOUT  (Shelly unreachable 30 min)
2026-03-13 09:33  TIER_3 -> LOCKOUT  (Shelly unreachable 31 min)
2026-03-13 13:50  LOCKOUT -> LOCKOUT (Shelly unreachable 30 min)
2026-03-19 13:54  TIER_3 -> LOCKOUT  (Shelly unreachable 31 min)
2026-03-20 16:10  TIER_3 -> LOCKOUT  (Shelly unreachable 31 min)
2026-06-23 08:39  TIER_3 -> LOCKOUT  (Shelly unreachable 30 min)
2026-07-21 17:45  TIER_3 -> LOCKOUT  (Shelly unreachable 30 min)

Seven for seven, the network. Never the pump. And each one cleared itself with a line like "Shelly recovered after 5.0 min stable," so even the state that requires a human has never actually waited for one. This is the state-machine version of something I found in the alerting a few weeks ago, where every urgent alert my monitor ever sent was about its own infrastructure rather than the pump. Same shape, two layers apart: the paths that reach for help are wired to the health of the monitoring, not to the health of the thing monitored.

The temperature doors are worth one more number. They have never fired, not once. But the hottest reading in the log is 59.4 °C, recorded on June 3rd at 1:37 in the morning with the pump pulling 511 watts in TIER_3. The soft lockout is 60.0. That door came within six tenths of a degree of opening, and 519 readings have been above the 50 °C warning line. The only escalation path with a real physical trigger has spent six months just barely not tripping.

What I'm Changing

Three of these are in the queue for the next time I restart the service on that box, which now stands at four pending one-line fixes. None of them is clever, and that is sort of the point.

1. Make the recovery test longer than the fault's return interval. Ninety seconds becomes thirty minutes of continuous idle, against a 34-minute median. If that feels long, that is the correct feeling: a recovery claim should cost more to make than the problem costs to recur. My criterion was about twenty-three times too cheap.
2. Count episodes, not cycles. Persist a climb counter across recoveries. The second climb within six hours should skip the staircase and go straight to the tier that the last one ended at. Re-deriving a known answer for two hours is not caution, it is amnesia.
3. Give the fault its own door to LOCKOUT. Total time in the top tier over a rolling window, or count of climbs per day. Something, anything, whose trigger is the persistence of the original condition. Right now that number is ninety percent of six months and it opens no door at all.
4. Stop calling it a success. "Confirmed unstuck!" is a claim I could not support, sent fifty-three times. Until the test can actually fail, the honest message is "pump idle for 90s, returning to NORMAL to observe."

As I write this, the monitor is in TIER_3. It has been there since 12:31 a.m. on the 11th, about thirty hours, and it is on cycle 151 of the same 120-seconds-on, 600-seconds-off pattern. The service has been up for 1,259 hours without a restart. Every individual piece is behaving exactly as I specified.

The pump, for its part, has never missed. It has run thousands of timed cycles at a healthy 490 to 505 watts, through whatever is actually wrong down there, and my basement has stayed dry the entire time. What I built on top of it is a machine that is very good at reaching the top of its ladder, and has no idea that the top of the ladder is where it lives.