My Pump Lost Power Three Times. I Logged It as a Reboot and Moved On.

In July 1997, a few days after Mars Pathfinder landed, the lander started resetting itself. A watchdog timer would notice that a critical task hadn't run, and force a total system reset. Each one cost a day of science. Hackaday retold the story this week, and the cause is the famous part: priority inversion. A low-priority weather task held a lock the high-priority data bus task needed, medium-priority work kept the low task from running, and the bus task starved until the watchdog pulled the plug. The operating system had a fix built in, priority inheritance, and the lock had been created with it switched off.

That's the part everyone teaches. The part I keep coming back to is three sentences earlier. Glenn Reeves, who led the flight software team, wrote afterward: "We did see the problem before landing but could not get it to repeat when we tried to track it down." Hackaday's summary is blunter: they chalked it up to glitches. And it quotes a rule from an old boss that I've now read four times: if you don't know why it happened, you don't know that it won't happen again.

So I went looking for the glitches I've chalked up. It took one grep.

Three reboots, zero causes

The smart plug my sump pump runs through reports a reason code every time it boots. Code 1 means POWERON: the device lost its supply and came back. Not a crash, not a watchdog, not a firmware panic. The outlet went dead and then it didn't. My monitor logs it like this:

[2026-10-05 22:24:19] PLUG REBOOTED: uptime 1721011s -> 29s — reset_reason=1 (POWERON (power-on))
[2026-10-05 22:24:49] WARNING: Plug output is OFF unexpectedly! Turning back ON.

Earlier in the year the plug rebooted constantly for firmware reasons I eventually chased down and fixed. Since the firmware settled in May, every single reboot has been a POWERON. There have been three.

I wrote about two of those three events at the time. The September one became a post about the plug's power-on default being wrong, which it is. The October one happened ten minutes after I finished repairing an eighteen-hour network outage, and the post was about the eighteen hours. In my own notes for that night it says "actor unknown." Three events, two write-ups, and in none of them did I ask the only question that matters about a POWERON: what cut the power?

What the log can say

Not much, but more than I'd used.

• The outages were short. At detection the plug's uptime was 29 to 39 seconds, and it answered the very next 30-second poll. Power was gone for less than a minute. A tripped breaker or a GFCI doesn't reset itself in sixty seconds, so unless someone was standing at the panel at 12:20 am, those are out.
• The monitoring server never lost power. It has been up since September 2. Either the house didn't blink and only that outlet did, or the house blinked and the server rode through on a battery. I don't know which, because I have run a 24/7 safety monitor for eight months without ever recording whether its host is on a UPS. I'll come back to that.
• Two of the three had no weather. I pulled hourly conditions for each. September 16 was clear and still, 12 km/h. October 5 was clear and dry but gusting 35 to 45 km/h all evening, which is the kind of night a utility line blinks. July 21 had a storm, but it had passed three hours before the reboot.
• One had a motor event right before it. Three minutes before the October outage, the pump drew 658 watts for three seconds and stopped. A single start transient. A loose receptacle plus a motor kicking against it is a hypothesis that fits a 30-second blip perfectly, and nothing in my log can rule it in or out.
• The October relay went off twice. After the plug came back, the monitor found the relay off, switched it on, and 30 seconds later found it off again. Either the plug rebooted twice more inside a minute, below the threshold my reboot check can see, or something else turned it off. I don't know. It's in the table now, which is more than it was yesterday.

Why the safe behavior is the problem

Here is what makes this uncomfortable rather than just untidy. The plug's power-on default is "off." I confirmed that in September and still haven't changed it. So every one of these blips leaves the pump unpowered until the monitor notices and switches it back on. With the monitor healthy, that's thirty seconds, and that's exactly what happened all three times. Nothing flooded. Nothing even got wet.

But on October 5 the monitor had just spent eighteen hours unable to reach the plug. If the blip had landed inside that window instead of ten minutes after it, the pump would have sat dark for the rest of the night with a monitor that couldn't see it and a default that wouldn't restore it. The thing that kept these three events harmless was timing.

Pathfinder's reset was correct engineering. Reeves is clear about it: "We knew that if this problem occurred we would reset." The watchdog did its job, the spacecraft survived, and that is also why the bug could hide through months of testing. My thirty-second auto-restore is the same shape. It works, and because it works I have never had a reason to look at what it was restoring from.

What JPL had that I don't

They found the Pathfinder bug on a ground replica of the spacecraft, with the trace facility turned on, running the same activity sequence over and over. Reeves: "In less than 18 hours we were able to cause the problem to occur." And the hooks that let them push the fix to Mars weren't an accident. His words: the debug features were "not 'fortuitously' left enabled but remain in the software by design."

I don't have a replica of my basement, and the one instrument I do have says only "power-on." A reset reason is a category, not a cause. It tells you the power came back. It doesn't tell you why it left.

What I can do is cheap, and it's the same move as every fix on this blog: add a second measurement the first one can't predict. A second plug, any Shelly or a ten-dollar power-loss logger, on a different outlet of the same circuit. Next time the pump's plug reports a POWERON, the monitor checks the second plug's uptime in the same breath. If both reset, it was the circuit or the utility, and the fix is a UPS or a conversation with the power company. If only the pump's plug reset, it's the receptacle or the plug itself, and the fix is a screwdriver. One extra device turns "reboot" into a word that means something.

And I'll go look at the server and find out whether it's on a battery. Eight months is long enough not to know.

The audit question

If you run any kind of monitoring on equipment you depend on, you have a version of this list. Every system that recovers automatically is building one.

List every event your system self-healed in the last ninety days. For each one, write down the cause. Count the blanks.

The blanks are not failures. They're the Pathfinder glitches: things the system handled so well that nobody had to understand them. Most of them will stay harmless. The one that doesn't will be the one you already saw three times.

For the record, the pit inspection I promised yesterday is still not done. It's day 229. Thirteen millimeters of rain fell on Monday and the pump's pulses didn't change at all, which is one more number pointing at the flashlight.