One Threshold, Two Jobs, 149 Cancelled Repairs

My sump pump monitor has exactly one mechanism for fixing things on its own. When it decides the float is stuck, it runs what I called an unstick sequence: cut power, then snap the plug on and off six times in about a minute, then leave it on for a while and see whether the pump stops by itself. It is a crude jiggle. It is also the only move the software has.

It has run 814 times since February. It has worked 5 times.

I have spent a week assuming that 0.61% success rate was a physics problem. A stuck float is stuck, and a smart plug that can only switch power is a poor instrument for unsticking it. That is a real limitation and I have written about it.

This week I went looking in the code for the rest of the explanation, and found something worse than physics.

149 of Those Attempts Quit on Their Own

Buried in the middle of the unstick loop, at the halfway mark, is a safety check:

if i == UNSTICK_CYCLES // 2:
    mid_status = get_power_status()
    if mid_status and mid_status["temp_c"] > TEMP_WARN_C:
        log(f"UNSTICK: aborting — temp {mid_status['temp_c']:.1f}C too high")
        turn_off()
        return False

Reasonable on its face. You are switching a five-hundred-watt inductive load through a small relay six times a minute. You should absolutely bail out if things get hot.

Then you look at what TEMP_WARN_C is, and what else it does.

Fifty degrees is my "getting warm, thought you'd like to know" number. It is the one that produces a chatty log line and a low-priority notification whose own text says, in as many words, that the temperature is not yet critical.

It is also, five hundred and forty lines away in the same file, a hard veto on the only repair the system can perform.

Here is what that cost:

Nearly one in five repair attempts terminated itself, halfway through, in a 1.1-degree band that starts one tenth of a degree above the trigger. Not one of them came within nine degrees of the threshold that does something real. The plug has never in its life been within eighteen degrees of the hard cutoff.

A constant that appears in two places has two owners and no owner. Whoever tunes it is optimizing for whichever consequence they can see.

I set that number for the log line. I could see the log line. I have never once seen the abort, because the abort is a line in a file I was not reading, in a routine that fails so often that one more failure is invisible.

The Part Where It Cancels Itself

The check fires forty-six seconds into the sequence, after three on/off cycles of a 490-watt load through the relay. So the obvious question is whether that extra half-degree was the basement being hot, or the routine heating its own relay.

I cannot fully separate those, because the code never takes a temperature reading before the sequence starts. But the clock says something. Here is when the aborts happened, by hour of day:

23:00 — 12 aborts
22:00 — 11
02:00 — 11
19:00 — 10
18:00 —  9
15:00 —  9
14:00 —  9
06:00 —  9

Attempts are spread nearly evenly across all twenty-four hours, so the abort rate is flat too. Eleven aborts at two in the morning is not what ambient summer heat looks like. It looks a lot more like a routine that generates the condition it then quits over.

Which is the structural lesson, and it does not depend on me resolving the causation:

The Same Number, Doing Its Other Job, 6,007 Times

The log-line half of this is its own small disaster.

That same 50 °C threshold has produced 6,007 temperature warnings since June 2nd, across 74 days. The worst single day was August 9th, with 327.

Three hundred and twenty-seven alarms in a day is 13.6 per hour. From one sensor. On one device. In one residential basement.

For scale: the published alarm-management guidance that industrial plants work to — EEMUA 191, and ISA 18.2 / IEC 62682 — puts the target under normal operation at roughly fewer than five alarms per hour for an entire operator, across an entire facility. My smart plug beat that by nearly threefold, on a single measurement channel, on a Sunday in August.

And I had written a latch specifically to prevent this:

if temp > TEMP_WARN_C and not sm.temp_warned:
    log(f"TEMP WARNING: {temp:.1f}C approaching limit ({MAX_TEMP_C}C)")
    send_notification("Sump pump: temperature rising", ...)
    sm.temp_warned = True
elif temp <= TEMP_WARN_C:
    sm.temp_warned = False

That code is correct. It fires once per rising edge and re-arms when things cool down. It is also completely useless here, and for a reason I should have caught the day I wrote it: the threshold sits inside the device's normal thermal oscillation.

The plug idles around 46 to 48 degrees. Every time the monitor runs the pump for its two-minute pulse, the relay warms past 50. During the ten-minute rest, it falls back below. Up, over the line, down, under the line. Every cycle, all day, for months.

So the latch re-arms every single cycle, and the "fire once" alarm fires 6,007 times. Hysteresis with a zero-width deadband on an oscillating signal is not hysteresis. It is a counter with extra steps.

Not one of those 6,007 warnings ever caused a human being to do anything. Since August 20th they have not even been delivered — they route through a notification path with an empty recipient list, so all of them land in the log as LOG email skipped: no recipients configured and stop there. Which, honestly, is the least harmful thing about them.

It Only Broke in Summer

Here is the detail that would have made this a customer problem instead of a blog post.

All 149 aborts happened between July 1st and August 26th. There have been zero since. Sixteen unstick attempts in the last three days all ran to completion.

Nothing was fixed. It is September. My basement is cooler, the plug now peaks at 49.6 °C during pump cycles instead of 50.5, and the gate simply is not being crossed anymore.

My recovery mechanism has a seasonal failure rate. It is weakest in July and August, which is precisely when heavy storm load makes a sump system most likely to need it. If I had shipped this to a customer, a spring pilot would have looked flawless and the thing would have started quietly failing in the summer, in a way that shows up in the logs as nothing at all — just a repair that ran a bit short and did not work, same as the ones that ran long and did not work.

Seasonal degradation of a safety mechanism is close to the worst failure shape there is, because the acceptance test and the failure mode never occupy the same month.

The Grep You Should Run This Week

The fix in my code is four lines. Split the constant into two:

TEMP_WARN_C          = 50.0   # log line only
UNSTICK_ABORT_TEMP_C = 55.0   # interlock; 5C below soft cutoff,
                              # 15C below hard cutoff, outside
                              # the plug's 46-51.5C normal band

— and sample it once, before the loop, instead of in the middle of it. That recovers eighteen percent of my repair attempts and deletes the seasonal failure mode.

But the fix that generalizes is not a number. It is a habit:

1. List every threshold constant in your system. Every _LIMIT, _MAX, _WARN, _THRESHOLD.
2. Count the references. Read every one over two.
3. For each additional use, ask: if I moved this number for the reason I normally move it, what else would change?
4. When a warning threshold and a control threshold want the same value, write them as two constants that happen to be equal. So the next person can separate them without archaeology.

That is an afternoon in a codebase you already own, and in my case it found a two-month-old defect that had disabled a fifth of my system's self-healing.

Why This Is a Building Problem, Not a Code Problem

I want to make the jump from my basement to your mechanical room explicit, because it is short.

Every building automation system I have ever opened has thresholds that were typed in once, during commissioning, by somebody who was tuning for how often the screen would go red. High discharge temp. Low suction pressure. Filter differential. Runtime alarm. A person picked those numbers, usually years ago, usually under time pressure, usually optimizing for exactly one visible consequence: the alarm list.

And some of those same numbers are interlocks. They lock out compressors. They inhibit stage-up. They cancel recovery sequences. They veto things.

Nobody has ever gone through and asked which is which.

Large plants have a name for this work: alarm rationalization. It is a formal discipline with published standards behind it, and process industries spend real money on it. Nobody offers it to a forty-thousand-square-foot building, because the consulting economics never worked at that size.

They work now. That is most of what has changed.

What I Actually Learned

This is the fourth day in a row that my own equipment has corrected something I believed about it. On Tuesday I thought the escalation ladder was welded shut. Wednesday it opened by itself for seven minutes. Thursday I found the exit condition was only being sampled during a sixth of each cycle. Today I found out that the repair I have been describing as ineffective has been cancelling itself in eighteen percent of attempts, over a temperature that means nothing, for two months, and that the reason it stopped is the weather.

Every one of those findings was sitting in a log file I have had since February. None of them required a better sensor or a bigger model. They required somebody to read what the system was already writing down and ask a question the alarm architecture structurally cannot ask.

The pump is still stuck, incidentally. Sixty-six hours in the top tier now, 326 cycles, and the float has not moved. Fixing the threshold would not have saved this particular lock — all sixteen attempts since Wednesday ran to completion and failed honestly. Sometimes the software really is out of moves, and the correct next step is a flashlight and a basement.

But now I know that for the previous two months, one attempt in five was not even a real attempt. That is not a physics problem. That is a number that had a second job I never gave it.