The Panics Stopped. Nothing Was Fixed.

I have spent about six weeks going through the logs of the monitoring rig on my own sump pump, in public, and writing down what I find. Most of what I have found has been in the soft layers. Mute logic that ate real alerts. An escalation ladder two hours tall. A blank line in a config file that silently beat a correct default and dropped an entire notification tier for two weeks.

There was a pattern in that, and I liked the pattern, because it was flattering. Every defect lived in the clever parts. The dumb part — the loop that decides to switch the pump on and then switches the pump on — had never once failed.

So this week I went looking for control-path failures, expecting to confirm that there weren't any.

The device that physically switches my pump is a $15 smart plug. It has rebooted 230 times. Two hundred and one of those were CPU exceptions.

The Number I Should Have Been Looking At

The plug reports why it restarted. Here is the whole six-month distribution, straight from the firmware's own reset_reason field:

A CPU exception is not the plug losing WiFi, and it is not the power flickering. It is the processor hitting a fault it cannot handle and cold-starting. Two hundred and one times, on the thing standing between my basement and a flooded floor.

I did not want to take the monitor's word for any of this, because the monitor is software I wrote and it has been wrong before. So I rebuilt the history from a different source: every heartbeat line the plug emits carries its own uptime, and I have 20,821 of them. Any time that number goes down instead of up, the plug restarted — whether or not my code noticed.

That reconstruction is what actually told the story, and it told a different one than the individual crashes did.

For three months that plug never ran six days. Not "rarely." Never. Across more than twenty thousand samples, 144.0 hours was the all-time high, and every approach to it ended in a CPU panic.

April shows it most clearly, because by then the chaos of March had settled into something with a rhythm:

Six consecutive crashes inside an eighteen-hour band, roughly every 5.9 days. That is not weather and it is not load. That is something filling up against a fixed ceiling and hitting it on a schedule. A leak, a counter, a buffer — I don't know which, and I'll come back to why that matters.

March was the same defect without the manners: 217 reboots, 63 of them after less than ten minutes of uptime. A hard crash loop, running from March 9th to the 20th.

Then It Stopped

Since June the plug has run 30 days, then 24, then 41. It is at 41.5 days as I write this, seven times past a wall it had never once crossed.

The obvious question is what fixed it. I checked the three candidates, and I checked them properly, because "something must have changed" is exactly the kind of assumption that has burned me before.

It wasn't a firmware update

I asked the plug directly. It is running version 1.3.3, build stamp 20240625 — compiled on June 25th, 2024. In the same reply it volunteers that stable version 1.7.5 is available.

The firmware has never been updated. Not before the panics, not during, not after.

It wasn't my code

Version control is unambiguous here. There are no commits to the monitor between March 8th and June 2nd. The entire regime change — crash loop, metronome, and then silence — happened inside a window where I changed nothing.

It wasn't a lighter workload

This was my best hypothesis and I liked it a lot. In early May the pump's float switch failed, my recovery routine gave up, and the monitor escalated to running the pump on a duty-cycle timer — which it has been doing ever since. I assumed that meant the plug was getting hammered differently and somehow never accumulating whatever it was accumulating.

The log says the opposite.

August works the plug about twice as hard as April did and has not crashed once. The busy months are the healthy ones. My hypothesis was not just unproven, it was backwards.

A reproducible defect has a fix. This one has a date it stopped and no explanation, which means the conditions that caused it were never identified and nothing prevents them from coming back. The plug is at day 41 of an uptime that used to be impossible past day 6, and I have no model that tells me whether day 42 is safe. "It seems fine now" is not an engineering position. It is a hope with a timestamp on it.

The Part That Actually Bothers Me

Here is what I keep turning over.

Of those 230 reboots, 229 sent me an email. They were delivered. They were correct. They came from the right machine at the right time and they said, accurately, that the plug had rebooted with a CPU exception.

My alerting worked. If you had asked me at any point this spring whether my monitoring was reaching me, I would have said yes, and I would have been right.

And I learned nothing from any of them, because a CPU panic arriving as one email at 3:22 on a Tuesday morning is an event, and the defect was never an event. The defect was a shape: a maximum that six months and twenty thousand samples never exceeded. No single message could have contained that finding. Every single message was true.

Event alerting answers "did something happen?" It cannot answer "is there a ceiling?" Those are different questions, and no amount of getting the first one right will answer the second.

This is the trap I want to hand to anyone running instrumentation in a building. The industry sells alerting, and alerting is genuinely necessary — but a system can have perfect alert delivery and still be structurally incapable of telling you the single most important thing about your equipment. Alerts are sampled at the moment of the event and then thrown away. Ceilings only appear when you keep everything and look at the distribution.

The Ten-Minute Version You Can Run This Week

You don't need my rig or my logs to do this. If you have any instrumented equipment at all, it is already reporting something that accumulates.

1. Pick a number that counts upward. Uptime, runtime hours, cycle count, free memory, signal strength. Anything that isn't a pass/fail flag.
2. Plot every sample you have. Not the last week, and not the alerts — the whole raw series, as far back as it goes. Six months if you can get it.
3. Look for a flat top. A value that repeatedly climbs toward a number and never passes it is a ceiling. Mine was 144.0 hours across 20,821 samples, and it was visible in about four seconds once the data was in one place.
4. A ceiling is always something. A counter rolling over, a leak filling up, a buffer, a timeout, a watchdog. It is never nothing. The value of finding it is that you get to ask what it is before it decides for you.
5. Then check the edges. If the flat part starts or stops, find out what changed. If the honest answer is "nothing," as it was for me, you now know that you don't understand your own system — which is far more useful than believing you do.

The whole exercise costs nothing. The data already exists. What is usually missing is that nobody ever puts it in one series and looks at the top of it, because everyone is busy reading the alerts, and the alerts are all arriving perfectly.

One More Thing, and It's Embarrassing

While I was digging through the repository for this, I found a file I wrote on February 27th: an over-the-air firmware upgrader for this exact plug. I built it, committed it, wrote a companion note titled "update firmware investigation," and then never ran it.

The device has been sitting on 2024 firmware ever since, through 201 CPU panics, with the vendor's fix path one call away.

I suspect this is the most common form of the problem in small operations. Not that nobody thought of the failure mode, but that thinking of it and building something felt enough like solving it. The check that catches this is dull and worth automating: for every mitigation you believe is in place, find the artifact that proves it ran — a version number, a timestamp, a log line. My belief was real. The version string said 1.3.3.

What I'd Tell a Building Owner

I am not going to pretend this reflects badly only on consumer hardware and not on me. But the honest framing for a client is straightforward, and it's the one I use.

The thing switching your pump, your air handler, or your walk-in cooler is very likely an inexpensive device running firmware nobody has thought about in two years. That is not a scandal; it is the actual condition of most small-building equipment, including the systems that work fine. The useful response is not to pretend otherwise or to sell you something expensive to replace it. It is three things:

• Know its failure modes from its own data, not from the spec sheet. The plug told me it had panicked 201 times. It had been telling me all along.
• Make sure it fails into a safe state. When mine crashed mid-sequence, the monitor detected the plug had dropped out and restored the correct switch position — 166 times out of 166. That recovery path is worth more than the crashes cost.
• Never let one $15 device be the only thing between water and a floor. Layers, and at least one that doesn't depend on software being right.

And keep every sample. Not because you will look at it often, but because on the day you finally do, six months of retained data will tell you something that six months of correctly delivered alerts could not.

More from this series: My loud alerts never stopped, so I missed 1,474 quiet ones | The self-repair routine ran 747 times | Everyone publishes capability, nobody publishes uptime | Watch the pump, don't drive it

Sources: all figures in this post are from my own deployment — a 141,548-line monitor log spanning 2026-02-23 to 2026-09-01, 20,821 device heartbeat samples, and live Shelly.GetDeviceInfo and Sys.GetStatus queries against the plug on 2026-09-01. Firmware version 1.3.3 (build 20240625), stable 1.7.5 available. Repository history via git log.