A Power Blip Checked the Setting I Promised to Check

At 00:18:40 last night the guardian process on my sump pump monitor read the smart plug the way it does every two minutes: relay on, 0.0 watts, 42.5 °C. At 00:20:02 the main monitor tried to read it and got nothing back. At 00:20:38 the plug answered again, and the first thing it reported was its own uptime: 31 seconds, down from 56 days. Reset reason: POWERON. The plug had lost mains power.

The second thing it reported was that its relay was off.

The monitor logged Plug output is OFF unexpectedly! Turning back ON. and turned it back on in the same second. The pump was idle the whole time. The pit was fine. From the last good reading to the relay closing again was under two minutes, and only about 31 seconds of that was the relay sitting open on restored power. As incidents go, it was nothing.

It is worth a post because of something I wrote exactly one week ago.

"I have not yet checked what mine is set to"

On September 9 I published "Leaving Plug in Current State" Is Not a Shutdown Plan, about what my monitor does to the relay when it exits. Near the end I pointed out that the plug has a hardware answer to that problem. Shelly Gen2 devices have a power-on default for the relay, on, off, or restore_last, and for a sump pump the right value is on. Then I wrote that I had not checked mine yet, and guessed that it was probably restore_last.

I still had not checked it when the power went out. So the power went out and checked it for me. This morning I asked the plug directly:

GET /rpc/Switch.GetConfig?id=0

{"id":0, "name":null, "initial_state":"off", "auto_on":false, ...}

off. After any loss of power, this plug opens the relay and waits for software to close it again.

Three documents said it was fixed

This was not a setting I had never thought about. It was a setting I had fixed, written up, and then trusted.

On February 23, a few hours after I first put this system in, a real power blip took the plug down. It came back with the relay off. The monitor caught it and turned it back on, and that evening I committed a lesson to the project README: check initial_state on any smart plug controlling critical equipment; set it to restore_last immediately after setup. The setup instructions got a step with the exact RPC call. A reboot investigation doc written that week says the switch reset is "mitigated by initial_state: restore_last." A firmware update script prints "Pump stays powered (initial_state: restore_last)" before it asks for confirmation.

That is three documents, all agreeing with each other, all written by me, and none of them is a reading. Each is a statement about what the device was set to on the day it was written. The device was never asked again.

It really was fixed, for a while

I did not want to replace one assumption with another, so I went back through every reboot this plug has logged since February and checked whether the relay came back off.

The July 21 row is the important one. The plug came back from a power-on reset with the relay closed, while the monitor was in a lockout state that sends no relay commands, and the guardian logged no action. Nothing I can find closed that relay. So in July, the power-on default was not off.

And the plug did not reboot again between July 21 and last night. Its 56-day uptime covers the whole gap. That leaves two possibilities: something changed the setting while the plug was running, or something turned the relay on in July that none of my logs can see. Nothing in the monitor's code writes that setting. I do not know which it was, and I am not going to guess in public twice in one post.

What I do know is that a setting can drift on a device that never reboots, and nothing in my system would notice until the next outage.

The February fix was the wrong value anyway

Here is the part that stings more. Even if the plug still said restore_last, it would not have been safe.

restore_last brings the relay back to whatever it was last told. My monitor has an escalation ladder for a pump that will not stop running, and its top rung runs the pump on a timer: 120 seconds on, 600 seconds off. I measured last week that the system has spent 89.9% of the last six months on that rung. So most of the time, the last command the relay received was off. Multiply it out and a power outage during a typical hour would have come back to an open relay about three times in four.

That is the 28 rows in March and April. Each one is restore_last doing exactly what it says, restoring an off. And 28 is a floor, because the monitor only logs "off unexpectedly" when it is in normal mode. The same restore during a timer rung looks like any other rest period.

The September 9 post already said this in one line: restore_last is just "inherit" wearing a firmware costume. I wrote the sentence and did not apply it to my own February fix. For a sump pump, on is the only power-on value that fails safe. The float switch is a perfectly good controller. Everything I built is supposed to be strictly better than the float switch. It is not better if a power outage turns the pump off.

What those 31 seconds were standing on

Last night went fine, so it is easy to read it as the system working. Here is what had to be true for the relay to close again:

1. Mains power comes back to the plug, and the plug boots (about 30 seconds).
2. The plug rejoins Wi-Fi, and the mesh node it talks to is already up.
3. The monitor host is up. It was: it did not reboot, so this outage never reached it. I do not yet know whether that is a separate circuit or luck.
4. The monitor process is running.
5. The monitor is in normal mode. This is the one that matters. The only code path in the whole system that turns an unexpectedly open relay back on lives in the normal-mode handler.

The guardian, my "independent safety backstop," will not help. Its only relay-on command fires when its own cooling rest ends. It never restores an off it did not cause.

Then look at step 5 during a long outage. If the plug has been unreachable for 30 minutes, the monitor goes into lockout, which sends no relay commands and waits for five minutes of stable contact before going back to normal. So the longer the power is out, the longer the software takes to restore the pump once power returns. For a sump pump that is exactly backwards. The longest outages leave the fullest pit.

One more detail from last night's log. The monitor did try to tell me. LOG email skipped: no recipients configured, twice, at 00:20:38. That is a known bug I have written about before and have not fixed. Today it also covered a power outage.

Meanwhile, at robot scale

Two robotics announcements this week happen to be about this exact problem, at a very different price point.

Agility Robotics unveiled Digit 5, the humanoid it plans to put next to warehouse workers with no fences. What caught my eye is not the legs. It is that the robot's safe state is physical: sit down into a stable posture and cut motor power before a person can touch it. And that response is enforced by an independent safety controller, not by the AI that decides what the robot is doing. Agility's co-founder Jonathan Hurst draws a line between certified safety and "a personal opinion about how safe you think a robot is." My three README paragraphs were a personal opinion about my plug.

InOrbit also open-sourced OpenRobOps, a fleet operations layer whose headline features include configuration as code (alert rules and telemetry config kept in Git) and a lock so only one operator commands a robot during manual intervention. Both are good ideas I do not have. But config in Git has the same weakness as config in a README. It is only true as of the last time something compared it against the machine.

You do not need a humanoid for this to apply. A smart-building newsletter this morning described a city whose grid-interactive pilot stalled because the building automation setpoints carried no consistent "reduce load" command. The city found that out only when it tried to use them. Same finding, bigger building.

What I am changing

None of this is applied as I write. It is the list, in order:

1. Set the power-on default to on. One RPC call, no restart, nothing else changes. It has been one RPC call this entire time, which I also said last week.
2. Make the monitor read it back. On startup and after every reboot the plug reports, fetch Switch.GetConfig and alert if initial_state is anything other than on. One extra request, and drift gets caught on the next reboot instead of the next flood.
3. Log recovery time as a number. Every power-on reset should produce one line: seconds from plug boot to relay closed. Last night I reconstructed 31 seconds from two logs and an uptime counter. It should just be written down.
4. Fix the three documents. Change them to say on, say why restore_last is wrong for a duty-cycled load, and say how to verify it.

The audit, if you run anything like this

1. For every relay, valve, or contactor on critical equipment, read its power-on state from the device today. Not from your notes, your install checklist, or your config repo. For a pump, a freezer, a sump, or a network switch, the safe answer is usually on. For a heater, it is off. Same relay, opposite answer.
2. If anything duty-cycles the load, treat restore_last as a coin flip. It restores whatever the controller last said, and a duty cycle spends most of its time saying off.
3. Count the layers between "power is back" and "the load is protected." If the count is more than zero, one of them will be missing during the outage that matters. The only zero-layer answer is the device default.

The basement is dry. The plug has been up for about seven hours. Its relay is on because software turned it on, and the setting that is supposed to guarantee that still says off. By the time you read this, that should be one RPC call closer to fixed. Then the monitor needs to keep checking it, because I have now shown twice that I will not.