Forty-Two Days at Maximum. The Exit Requires a Crash.

Yesterday I wrote that the smart plug controlling my sump pump had CPU-panicked 201 times, held a hard six-day uptime ceiling for three straight months, and then quietly stopped doing it with no firmware update, no code change, and more load than before. I called that worse than a reproducible bug, because a bug that stops without being fixed leaves you with no model of when it comes back.

Today I went looking at what those panics had been doing while they happened.

They had been resetting my control loop. And now that they have stopped, my system has been sitting in its maximum emergency state for forty-two days.

What the Machine Looks Like

The monitor is a state machine. When the pump runs continuously past a threshold, that usually means the float switch is stuck closed, so the software steps through a ladder designed to protect the motor: try a power cycle, then duty-cycle the plug on progressively harder schedules.

I am reasonably proud of this design. It escalates gradually, it protects the motor from continuous running, it never leaves the basement unprotected, and it worked exactly as intended for months.

The last time it changed state was July 21st at 11:14 PM.

Forty-Two Days

Every state transition my monitor has ever made is written to the log with the word STATE in it. There are 352 of them across six months. The most recent one is six weeks old. Here is what has happened since:

One hundred and sixty-eight hours of run time. Five thousand starts. On a motor whose duty rating assumes it runs when there is water and stops when there isn't.

The Exit Conditions

I did not want to guess at why it was stuck, so I read the code rather than the logs. There are exactly two ways out of TIER_3.

The first is a mid-pulse stop. While the plug is energized, the monitor polls every five seconds. If more than ten seconds in the pump is no longer drawing power, the float has evidently released and the machine goes to COOLDOWN, then back to NORMAL if the pump stays idle. This path works. It has fired 65 times.

The second is a dry streak. At the end of a pulse, if the pump is drawing under 100 watts, a counter increments. Three consecutive dry pulses means the water table is low and the machine returns to NORMAL.

The second path is effectively dead code, and the logs prove it. Because the mid-pulse check runs on every poll from second 10 through second 120, any pulse that goes dry gets caught by the first path long before the second one is evaluated. For the dry counter to increment, the pump has to still be pumping at second 115 and dry at second 120. That is roughly a one-in-twenty-four window, and you need it three times in a row.

So in practice there is one exit, not two, and both are conditioned on the same physical event: the pump stops drawing power while the plug is energized.

In 5,040 consecutive attempts, my pump has not come within 348 watts of that.

What Actually Reset It, the Last Two Times

Here is where yesterday's post and today's collide.

I pulled the full transition history and bucketed it by type. The ladder has been reset by exactly three mechanisms in six months: the float physically unsticking, the plug going unreachable for thirty minutes, or the plug rebooting and coming back.

The last three times my system escaped TIER_3 were June 18th, June 23rd, and July 21st. The June 18th one was a genuine float release. The other two were both logged as Shelly unreachable 30 min — the monitor losing contact with the plug entirely, locking out, and then re-entering the ladder from the bottom when the plug came back.

The July 21st event is in the plug's own words:

PLUG REBOOTED: uptime 2089696s → 39s — reset_reason=1 (POWERON)

That is a mains power event. Not a panic, not a software action, not anything I did. It is also the only plug reboot in my log since April.

Put the two halves together. Back in March, when this plug was crash-looping and rebooting 217 times in a month, my escalation ladder was being reset constantly. Every crash dumped the state machine back to NORMAL and made it climb again. I never noticed, because I was never stuck.

I want to be careful here, because there is a version of this story that is too neat. TIER_3 is not a malfunction. It is the correct, protective response to a stuck float, and running a pump on a 120-second duty cycle is much better than running it continuously. Every individual component of my system did precisely what it was specified to do for forty-two days.

The defect is not in any of the parts. It is that nothing in the design distinguishes forty-two minutes in the emergency state from forty-two days.

The Missing Input Is a Clock

Every transition in my state machine is conditioned on the equipment. Is the pump drawing power. Is the plug reachable. Is it too hot. Those are the right questions and they are correctly implemented.

Not one of them asks how long we have been here.

A state with no time-based exit is not a state. It is a destination.

The information was never missing. My monitor computes exactly how long it has been in the current state, on every single loop iteration. It uses that number in a status line for two of the seven states and discards it everywhere else. The sentence "this system has been at maximum emergency duty for six weeks" is one my software has had all the facts to say, five thousand times, and has never said once.

It Was Talking the Whole Time. Quietly.

This is the part that connects to everything else I have written in this series.

My monitor was not silent about any of this. In forty-two days it generated 8,785 notifications, roughly one every seven minutes:

Every one of them was sent at the routine tier. The urgent tier, which is the one that reaches my phone, sent zero messages during the entire forty-two-day run.

And the classification is defensible message by message. One TIER_3 cycle notice genuinely is routine. The code's own escalation email says so in plain text: "This is designed escalation behavior, not urgent." That sentence is true.

It is true of the first one. It is not true of the five-thousandth. But urgency in my system is a property assigned to a message, and the condition I needed to know about is a duration, and no amount of correctly classifying individual messages will ever produce that.

Yesterday's version of this lesson was that event alerts cannot describe a distribution. Today's is narrower and more actionable: event alerts cannot describe a duration either. Both are properties of the set, not of any member of it.

Three Checks Worth Ten Minutes

You do not need my rig to run any of these. If you have a BMS, a PLC, a controller, or anything with a mode, these apply.

1. Count your state transitions by type. One pass over your logs, bucketed. Anything at zero after six months is either dead code or an exit you have never tested. Mine had a recovery path with a lifetime count of one, and I would have sworn it was working, because it is right there in the source and it reads correctly.
2. Put a ceiling on time-in-state. Pick a number of hours past which any non-normal state escalates to a human, and make that escalation loud regardless of what the underlying sensor says. This is a small patch. In my case it is a few lines next to a variable I already compute.
3. List what has actually reset your system in the last six months. Not what is supposed to reset it. What did. If the honest answer includes a crash, a reboot, a power blip, or an outage, then your recovery depends on failures, and improving your hardware will break your software in a way no test will catch.

What I'd Tell a Building Owner

Almost every equipment problem I get called about is described as a detection failure. Very few of them are.

Your building management system probably already knows that a valve has been in manual override since spring, that an air handler has been running on its backup schedule since a technician visit in June, that a zone has been calling for heat and not getting it for eleven weeks. Those are not undetected conditions. They are detected conditions that nobody ever asked the duration of, and duration is the thing that turns a correct temporary response into an expensive permanent one.

I read a story this week from an enterprise assessment walkthrough: a team traced a mystery cable into a wall, cut the sheetrock, and found the building's original time clock, still wired to a phone line somebody had been paying for twenty years. Nobody had dared disconnect what nobody understood.

That is a twenty-year version of my six weeks, in a building a thousand times the size of my basement, and it happened for the same reason. No process asked how long.

So the question I would put to any owner, and the one I now have to answer about my own equipment: which of your systems is currently in a mode it entered as a temporary measure, and how long has it been there? If your monitoring cannot answer that in one query, it is not answering the question you actually needed answered.

Mine could not. It has all the data. It just never thought to look at a clock.

More from this series: The panics stopped, nothing was fixed | The escalation ladder was two hours tall | The self-repair routine ran 747 times | My loud alerts never stopped, so I missed 1,474 quiet ones

Sources: all figures in this post are from my own deployment — a 142,478-line monitor log spanning 2026-02-23 to 2026-09-02, its 352 state-transition records, 6,004 device heartbeats since 2026-07-21, the monitor source (1,334 lines), and a live Shelly.GetDeviceInfo query against the plug on 2026-09-02. The enterprise time-clock anecdote is from John DeVeaux of Hines, related in the Nexus Labs "Owner Signal" newsletter of September 2, 2026.