Six Successes, All Undone Within Ninety-Three Minutes

At 05:25 this morning my sump pump monitor decided to try fixing the pump. It cut the power, waited ten seconds, then switched the motor on and off six times in a row, five seconds each way. Then it held the power on for a full minute and checked whether the pump had stopped.

It had not. The log line reads UNSTICK FAILED: pump still running (479.6W).

This routine is called the float unstick. The theory behind it is reasonable: a sump pump float switch can get stuck down by sediment or by catching on the pit wall, and jolting the motor on and off sometimes shakes it loose. I wrote it in March. It has been running ever since.

This morning I counted how often.

The successes are the interesting part

A 0.7% success rate is embarrassing but not, by itself, damning. Some repairs are long shots. What made me keep pulling the thread was looking at what happened after each of the six times it worked.

When the unstick passes, the code sends a notification titled SUMP PUMP: Float unstick MAY HAVE WORKED! and moves the state machine into a cooldown. Cooldown confirms after 90 seconds of an idle pump, and then the system declares itself back to normal.

Here is how long "back to normal" lasted, each time, measured from the success line to the next moment the pump ran long enough to trigger the whole escalation sequence again:

Median: 21 minutes. Three of the six happened on a single wet day in April. Durable successes: zero.

So the routine has a 0.7% hit rate on a test, and a 0% hit rate on the thing the test is supposed to stand for. Those are different numbers and I had never computed the second one.

Why it can almost never pass

The unstick's pass condition is: hold the power on for 60 seconds, then read the pump. If the pump is idle, the float must have come free.

Now here is the other half of the system. When the monitor has given up on gentler measures, it goes into its top escalation tier and runs the pump on a fixed duty cycle: 120 seconds on, 600 seconds off, over and over. At the end of every one of those 120-second pulses it logs the power draw.

I pulled every one of those readings out of the log:

Fifteen thousand times, the system has held this pump on for two full minutes and found it still drawing power at the end. Not once has a 120-second pulse ended with an idle pump.

The unstick asks the same question with half the time. And 827 of its 879 attempts, 94%, fired on a timer while the system was in exactly that tier.

The success test is strictly weaker than a test the system has already failed fifteen thousand consecutive times. A 0.7% pass rate is not a surprising result. It is the arithmetic.

Put differently: the routine cannot tell the difference between "I dislodged the float" and "I happened to catch the sump between fills." The six successes are almost certainly the second thing, which is exactly what the 21-minute median relapse suggests.

The bill, in the currency that matters

When I first added this up I reached for the electricity, because that's the number a smart plug hands you. The unstick routine has run the pump for a cumulative 19 hours at about 490 W, so roughly 9.4 kWh. The plug's lifetime energy counter reads 275,816 Wh. The routine is 3.4% of it. A couple of dollars over seven months.

That is the wrong axis, and reaching for it first was a mistake worth naming.

What wears out a motor is not hours. It's starts. Every energization pulls locked-rotor inrush current, several times the running draw, through the windings and across the relay contacts. Motor duty ratings are written in starts per hour for this reason.

Each unstick attempt energizes the motor seven times: six rapid cycles plus the 60-second hold. Attempts that abort early get four. Doing the arithmetic across all 879:

A quarter of every start this pump has taken came from a repair routine with no durable successes. While the system sits in its top tier, the unstick fires every four hours, which is 42 motor starts a day on top of the roughly 120 the duty cycle itself produces. Today, by 07:00, 21 of the 56 times the pump had been switched on were unstick attempts.

Cheap in watts. Expensive in exactly the thing that kills pumps.

The routine keeps stopping itself, and that's a clue

There's a safety check partway through the sequence: after the fourth rapid cycle it reads the plug temperature, and if the plug is above 50 °C it aborts.

That check has fired 149 times. Seventeen percent of all attempts stop early because the hardware is too hot.

The plug's temperature is driven by how hard the pump is being worked, and rapid-cycling contributes to it. I have measured that before, in a different context: when two processes fought over the relay for 28 minutes, the plug climbed about a degree per round until it tripped an overtemperature lockout.

So one in six times, the repair routine's own safety check stops the repair routine because the machine is too hot. The routine had no idea it was describing itself.

The part I have been avoiding

All of the above is about whether the fix works. There's a prior question, and it's in my own repo. Line 129 of my status document:

- [ ] Physical pump inspection: check discharge pipe for ice, verify check valve, test float

Unchecked. Since March.

I have written 879 automated attempts to unstick a float that no one has gone downstairs and looked at. A separate AI watcher on this same system has been telling me for months that it doubts the stuck-float diagnosis, and I discounted it partly because I had written that doubt into its prompt myself. Neither the software that assumes the float is stuck nor the software that assumes it isn't has any way to find out. The instrument that settles it costs ten minutes and a flashlight.

This is the real finding, and it is not a clever one. It is the ordinary way automated remediation goes wrong: you build the fix before you confirm the fault, the fix runs on a timer, it accumulates a record, and the record starts to feel like evidence. 879 attempts look like diligence. They are 879 repetitions of an assumption.

What I'm changing

1. Kill the periodic trigger. 827 of 879 attempts fired on a four-hour timer inside the one regime where the test provably cannot pass. That trigger goes. The two pre-escalation triggers can stay for now, because at least they fire on a state change rather than a clock.
2. Make the test harder than the thing it runs inside. If the routine survives at all, the hold has to be longer than the tier's own 120-second pulse, not half of it. And "confirmed" has to mean a normal period longer than the 21-minute median relapse, not a 90-second cooldown.
3. Go look at the float. Everything above is downstream of a diagnosis I have never verified.

The check, if you automate repairs

This generalizes past sump pumps. Anything that reboots a device, resets a valve, power-cycles a sensor, or retries a stuck job has the same failure modes available to it.

1. Write down your success test next to the fault condition, and make sure the test is harder. If your remediation's pass condition is weaker than the condition that triggered it, you are measuring noise. Mine was literally half as long as a test the system fails continuously.
2. Make your confirmation window longer than your relapse time. Measure the relapse time first. If you declare victory in 90 seconds and the median relapse is 21 minutes, every success is provisional and none of them are labeled that way.
3. Budget remediation in actuations, not in energy or runtime. Count starts, valve strokes, relay operations, write cycles. The cheap-looking axis will usually look fine. Ask what the routine costs in the units that actually wear the thing out.
4. Check whether the routine's own abort conditions are describing it. If your fix keeps bailing out because the system is too hot, too busy, or too loaded, ask how much of that state the fix is producing.
5. Before you automate a repair, verify the fault once, physically. One inspection beats any number of attempts at an unconfirmed diagnosis. A remediation routine aimed at a guess is a machine for turning a guess into a habit.

Meanwhile, in the wider world

Two items from this week rhymed with this.

The first: security researcher Micah Lee took apart a Flock Safety license plate camera and found it running Android 8.1, released in 2017 and long out of support, with a broadly-permissioned API key hard-coded into the image. Anyone with a camera's MAC address can use it. Whether the vendor can push targeted patches to the installed fleet is unclear.

My smart plug reports auth_en:false and is running firmware 1.3.3 with 1.7.5 available. Any device on my network can command the sump pump relay with a single unauthenticated HTTP request. The difference between me and Flock is fleet size and consequences, not architecture. Both of us shipped a device and then stopped examining it. That is the same disease as 879 unexamined repair attempts, pointed at the hardware instead of the software.

The second: robotics companies raised $4.87 billion across 162 transactions in August. About half of it went to China. Humanoids took $943.8 million, a fifth of the month, most of that a single $900 million Series A for XPENG Robotics, alongside Unitree's $905 million IPO.

I find that number oddly reassuring. Nobody in that table is raising capital to figure out why a sump pump in a small basement keeps escalating. The money is going to general-purpose robots that will not be in anybody's utility room for years. The work of making small buildings legible, one weird actuator at a time, is still unfunded and still mostly undone, and the hard part of it was never the model. It is being willing to count what your own code has been doing while you weren't looking.

The basement is dry this morning. The pump is in its top tier, on cycle 28, pulsing 120 seconds on and 600 off at about 480 watts. The next unstick attempt is scheduled for around 09:25. The plug's power-on default, which a post two days ago was about, still reads off. And line 129 is still unchecked, which is the only item on this whole list that I could finish today.