"Leaving Plug in Current State" Is Not a Shutdown Plan

My sump pump monitor writes one line when it exits. I wrote it myself, months ago, and I have read past it a hundred times without seeing it:

[2026-06-23 07:35:10] Shutting down monitor. Leaving plug in current state.

Leaving plug in current state. It sounds careful. It sounds like restraint, like the software declining to meddle on its way out the door. It is actually the most dangerous line in the program, and it took me six and a half months and a wrong query to notice.

The thing in series with the pump

Some context on the setup. There is a sump pump in my basement. Its float switch is unreliable, so I put a smart plug between the pump and the wall, and a small Python service on a machine upstairs that watches the plug's power draw and decides when the pump should run. When the float sticks, the service falls back to running the pump on a timer, which is what it has been doing continuously since September 2nd.

The important structural fact, the one I managed not to think about for half a year: that relay is in series with the pump's power. The monitor does not observe the pump. It controls the pump. Everything downstream of that plug is available to my software to switch off, including the pump's ability to remove water from my house.

In its current mode the duty cycle is 120 seconds on, 600 seconds off. Run the arithmetic on that and you get the number this whole post turns on: the pump is unpowered about 83% of the time by design. That is not a fault. On a healthy day, if you freeze the system at a random instant, five times out of six the relay is open and the pump is dark, waiting for its next pulse.

Now put those two facts together with the exit handler. The resting state is off. The exit rule is "leave it where it is." Which means that any unplanned stop of the controller has, roughly, a five-in-six chance of freezing the pump in the off position, with nothing left running to ever turn it back on.

Fifteen out of fifty-seven

So I went and counted. Every clean shutdown in the log, sorted by what the last command to the relay had been.

Fifteen times, the software let go of the relay while the relay was open.

Twelve of those fifteen are nothing. They are systemctl restart cycles, and the service came back in the same second or inside a minute. A pump that misses one 120-second pulse has missed nothing that matters.

Three were not nothing.

The 28-day one is a planned hardware move — the plug's IP address changes across the gap — and I have no record of how the pump was powered during it. I am not going to claim my basement went a month without a working pump when the honest answer is that I do not know. That leaves June 23.

June 23, three hours

June 23 is a date I already wrote about, for a different reason. It is one of the two occasions when my system's loudest alert fired and the message never left the building, because the network fault it was reporting was also the network fault the email needed.

Here is that morning with the relay commands laid in alongside it.

[07:28:06] Plug turned OFF          ← normal end of a timer pulse
[07:35:10] Shutting down monitor. Leaving plug in current state.
           ... 33 minutes. No controller. Relay open. ...
[08:08:41] === Sump Pump Monitor Started ===
[08:39:03] SHELLY CRITICAL → alert LOST (email and push both failed)
[10:10:58] Shutting down monitor. Leaving plug in current state.
           ... 23 minutes. No controller. Relay open. ...
[10:33:43] Monitor ready: State: LOCKOUT | Reason: Shelly unreachable 30 min

Three failures in three hours, from three unrelated parts of the system. The controller exits and abandons the relay in its off phase. The network then degrades until the plug is unreachable entirely. And the alert that exists precisely to tell me about that dies of the same network fault.

None of the three knows the other two exist. There is no line of code anywhere in my system that could have noticed them stacking.

And the detail that makes it land: the float switch was stuck that whole morning too. The pump was not running on its own float. The timer was the only thing pumping water out of my house. The timer was the thing that stopped.

The direction of safe

Here is the design idea I was missing, and it is not a sophisticated one. It is the first thing you learn about relays and the last thing you remember about your own code.

Anything that can remove power needs a defined state for its own absence. Not an inherited state. A chosen one.

My exit handler does not choose. It inherits — from whatever the duty cycle happened to be doing at the moment the process received its signal. Inheritance is a fine default for a lot of software. It is a terrible default for an actuator, because the thing you inherit from was optimizing for something else entirely.

For a sump pump, the safe direction is energized. A pump plugged straight into a dumb outlet with a flaky float is the situation I had before I built any of this, and it was survivable for years. My monitor is supposed to be strictly better than a dumb outlet. An exit path that can leave the relay open is strictly worse than a dumb outlet, in exactly the circumstances where I am least likely to be watching.

Notice that the answer is load-specific and reverses cleanly. For a space heater, the safe direction is de-energized — a heater stuck on with nothing watching it is the fire. Same relay, same firmware, same $30 device, opposite correct answer. And it ships with one default for both.

Two fixes, neither of them clever

One: the exit handler should command the relay on before it exits, and log what it did rather than logging that it declined to act. That is a few lines.

Two, and this is the one I feel worse about: the plug already does this in hardware and I never configured it. Shelly Gen2 devices expose a power-on default state — on, off, or restore_last — along with auto-off timers. Set the default to on and the relay fails safe even when the controller, the host it runs on, and the network are all gone at once. It is one RPC call. It has been one RPC call this entire time.

I have not yet checked what mine is set to. The LAN dropped out from under me while I was writing this, which is its own small joke. But restore_last is a very common factory default, and restore_last is just "inherit" wearing a firmware costume.

A note on how I nearly got this wrong

I want to include this part because it is the more useful lesson.

My first pass at this analysis reported a 73-day outage and a second 28-day one. Both were fabrications, produced by my own script. I was matching each shutdown to the next restart with a binary search that used bisect_right, which skips exact matches — and the overwhelmingly common case is a restart landing in the same second as the shutdown that preceded it. So every ordinary service restart looked like an abandonment lasting until whenever the next one happened to be.

The log was correct. The query was wrong. I caught it because the same timestamp appeared as the start of ten different "gaps," which is not a thing that happens in the physical world.

This is the third time this particular log has punished me for inferring from the shape of a gap rather than reading what is actually on either side of it. The fix has been the same all three times: open the file and read the adjacent lines. A gap is not evidence. A gap is an absence of evidence that is very easy to mistake for the other thing, especially when it confirms a story you have already started telling.

The related trap, from the same morning: my log contains 171 lines reading === Sump Pump Monitor Started ===. Ninety-eight of them are immediately followed by ERROR: Another instance is already running. Exiting. There have been 73 real starts, not 171. Anything that counts start banners overcounts restarts by more than double.

I publish these while they are still embarrassing, because a monitoring system whose failure modes are undocumented is not a product. It is a hope with a subscription attached.

My honest answers today: my relay holds, my device default is probably "restore last" and I have not confirmed it, and nobody decided anything. The duty cycle decided, and the duty cycle's opinion is "off, five times out of six."

The pump has now been running on a timer for seven days straight, 801 cycles, every one of them drawing full power, because a ten dollar float switch is stuck. The system is handling it. Today I learned that the system's plan for its own death was to stop handling it and leave the relay open.