The Relay Got Hot Enough to Cancel Its Own Repairs

Yesterday I counted the outcomes of my sump pump's automatic repair routine for the first time in five months and found 747 attempts and four successes, the most recent of them in April. This morning I added up three numbers I had printed the day before and never summed.

Here they are, refreshed, along with a fourth line I had left out:

$ grep -c "UNSTICK ATTEMPT"   sump_pump_monitor.log
753
$ grep -c "UNSTICK FAILED"    sump_pump_monitor.log
601
$ grep -c "UNSTICK: aborting" sump_pump_monitor.log
148
$ grep -c "UNSTICK SUCCESS"   sump_pump_monitor.log
4

601 plus 148 plus 4 is 753. Every attempt this system has ever made lands in exactly one of those buckets, which is satisfying, and then you notice which bucket is second-largest.

The Shape Was Wrong

My first assumption was that aborts were a steady background rate, a few percent of attempts, uninteresting. So I broke them out by month, expecting a flat line.

2026-03     0
2026-04     0
2026-05     0
2026-06     0
2026-07    67
2026-08    81

Nothing for four months. Then 148 in eight weeks. That is not a rate, that is an event. Something crossed a line in July and has stayed across it since.

The abort has exactly one trigger, forty lines into the repair routine:

if mid_status and mid_status["temp_c"] > TEMP_WARN_C:
    log(f"UNSTICK: aborting — temp {mid_status['temp_c']:.1f}C too high")

TEMP_WARN_C is 50.0 degrees. Halfway through shaking the float loose, the system takes a reading, and if it is above fifty it stops and walks away. Reasonable enough as written. So I went to look at how badly the equipment was overheating, and found this:

[2026-08-19 23:25:19] UNSTICK: aborting — temp 50.7C too high
[2026-08-20 03:25:37] UNSTICK: aborting — temp 50.1C too high
[2026-08-20 11:28:13] UNSTICK: aborting — temp 50.7C too high
[2026-08-20 15:28:27] UNSTICK: aborting — temp 50.3C too high

50.7. 50.1. 50.7. 50.3. Nothing is overheating. Those are the readings of a number that has come to rest on top of a threshold and is now scraping against it every few hours.

Whose Temperature Is That, Anyway

This is the part where I have to admit to five months of reading my own logs wrong.

Every cycle in that file prints like this: TIER_3 cycle 4208: pump ran (503.1W, 48.6C). Watts and degrees, side by side, describing a pump. For five months I read that as the pump is at 48.6 degrees, because that is what the sentence looks like it says.

It is not what it says. One line in the status parser:

"temp_c": sw.get("temperature", {}).get("tC", 0.0),

That is the smart plug reporting the temperature of its own circuit board. Not the motor. Not the water. Not the pit, not the discharge pipe, not the basement air.

I want to be fair to the design here, because this was not a stupid decision, it was a free one. The plug exposes that reading over its API. It costs nothing to poll. It is genuinely useful for what it is actually for, which is catching a relay that is arcing or a plug that is failing. What happened is that a free signal in roughly the right place got quietly promoted into a proxy for the health of the load, and nobody in the loop, including me, wrote down that it was a proxy.

Five Months of the Switch Warming Up

Once you know whose temperature it is, the obvious question is what it has been doing. I pulled only the idle readings, the heartbeat lines logged when the pump is drawing 0.0 watts, so nothing is running and this is purely the plug sitting there at rest.

The idle temperature has climbed 14.4 degrees since March. The power draw has gone down about five watts over the same period.

So this is not a pump working harder. Whatever is making heat is making it while the pump is off, and it scales with the fifth column, not the fourth. Switching a 500-watt inductive load is where a smart plug generates heat, and my commanded switching went from 1,835 operations in March to roughly 4,500 a month from June onward. The relay never gets a long enough rest to come back down, so the floor it starts each cycle from keeps creeping up.

And the warning line is 50.0 degrees. The plug now sits at 48.3 doing absolutely nothing. The entire remaining margin between idle and alarm is 1.7 degrees, which a single 120-second pump cycle closes without difficulty.

You can watch that happen in the alarm counts. TEMP WARNING fired 41 times in June, 2,568 times in July, and 3,308 times in August. Nothing about the equipment changed between June and July. The floor rose into the alarm band, and the alarm has been sitting in it ever since.

The Loop

Here is the whole thing, and it is a circle:

1. The float switch stuck. The system escalated to its maximum duty cycle, 120 seconds on and 10 minutes off, and began commanding the relay about 4,500 times a month instead of 1,835.
2. That switching volume warmed the relay. Idle temperature drifted from 33.9 to 48.3 degrees over five months.
3. The new floor pushed readings under load through the 50-degree line as a matter of routine.
4. That same 50-degree constant gates the repair routine. So from July onward, the relay's accumulated heat started cancelling repair attempts partway through. Sixty-seven in July, eighty-one in August. Forty-four percent of every repair attempt in the last two months, 148 of 335, was called off by the system's own switching heat.
5. Every one of those cancellations was written to a notification channel that has had no recipients since the twentieth.

The system's compensation for the fault is now the main thing preventing the fix. And nothing inside it can perceive that, because the only instrument it has is inside the loop it would need to observe.

Why It Never Steps Back Down

At this point I went looking for the exit, on the theory that a system this thoroughly stuck must have some path back to normal that was simply not being taken.

Here is every state transition in 135,989 lines of log, five months of continuous operation, grouped by kind:

76  NORMAL      -> POWER_CYCLE
57  POWER_CYCLE -> TIER_*
49  COOLDOWN    -> NORMAL
19  POWER_CYCLE -> NORMAL
12  COOLDOWN    -> TIER_*
 5  LOCKOUT     -> NORMAL
 1  LOCKOUT     -> LOCKOUT

Read that list for what is missing. There is no TIER_3 -> TIER_2. There is no downward transition between tiers of any kind, anywhere, in five months.

There is no such transition because there is no such code. Each tier in the config table has a next and an escalate_after. Grep fifty-seven kilobytes of monitor source for de-escalate, or deescalate, or a prev key, and you get nothing at all. Every one of the eight matches is next_tier. I built a ladder with rungs going one direction and did not notice for half a year, because until the float stuck I had never needed to climb down it.

There is one exit from the tiers, COOLDOWN -> NORMAL, and cooldown has a single entry condition: the pump stops on its own during an on-pulse. Which is a thing a stuck float definitionally prevents.

Which produces this, the last line in the log where anything about this system's condition changed:

[2026-07-21 23:14:51] STATE: TIER_2 -> TIER_3 (escalation from TIER_2)

That was thirty-five days ago. Since then the cycle counter has gone from 1 to 4,209, at maximum duty, continuously, and there is no mechanism anywhere in the software capable of concluding that anything has changed. The last time it entered cooldown at all was June 18. The last time it reached normal was June 15.

And the One Channel That Still Works Says This Is Fine

The daily digest survived this month's notification outage because it runs on its own timer with its own hardcoded recipients. Here is what it said yesterday morning:

HIGH_INFLOW INFO | shelly=192.168.68.109 0W 45.7C OFF state=TIER_3 | Pumping a lot but pump is healthy (8 min/hr)

"Pump is healthy." It prints state=TIER_3 in the same line and draws no conclusion from it whatsoever, because "how long have we been sitting in the terminal state" is not a quantity it computes. Thirty-five days pinned at maximum. A relay 1.7 degrees from its own alarm floor while idle. A repair routine at four-for-753 with 44 percent of recent attempts self-cancelled. Severity: INFO.

And in fairness to it, every individual word of that line is true.

Four Things That Generalize

1. Your only sensor is often inside your actuator

A smart plug measuring its own die temperature is not a proxy for the equipment. It is a proxy for how hard you have been switching it. Any threshold you set against it is really a threshold on your own control behavior, and it will drift as that behavior drifts. Mine did, by 14.4 degrees, in the direction that matters.

The rule I would write on the wall: if a sensor is physically part of your control path, it cannot independently validate your control decisions. If the same reading gates both a safety interlock and a remediation action, you have wired your repairs to your own duty cycle and the failure mode will look exactly like mine, which is to say invisible and slow.

2. Escalation ladders need rungs going down, with their own trigger

Not the inverse of the up-trigger. Its own. "Stopped on its own during an on-pulse" cannot detect recovery from a fault whose entire nature is preventing the pump from stopping on its own. When the only de-escalation path is gated on the fault being absent, and the fault is what suppresses the signal, you have built a state machine with a terminal state that you did not intend to have and will not find until you are living in it.

This is worth checking in anything you run that escalates: alerting, retries, rate limiting, thermal management, autoscaling. Write down every state and ask what specific observation moves you out of it. If you cannot name the observation, or if naming it requires the problem to already be over, that state is an absorbing one.

3. Thresholds are calibrated against a baseline, and baselines move

Fifty degrees was a defensible number in March. The floor was 33.9, so it represented a sixteen-degree margin and a genuine anomaly. Against August's floor of 48.3 the identical constant represents 1.7 degrees and fires three thousand times a month. Nobody edited it. Nobody had to. The number stayed still while the world moved under it, which is the ordinary way that a good alarm becomes a bad one.

The fix is not a better constant, because a better constant expires too. It is to define the alarm relative to a measured trailing baseline, so that it means the same thing in month six as it did in month one.

4. Time-in-state belongs on the front page

Every dashboard in this incident is accurate. Every reading is in range. Nothing is failing, in the sense of any individual value being wrong. The number that would have caught all of this, and that nothing anywhere prints, is thirty-five days.

For any system with a state machine, the duration in the current state deserves to sit directly beside the state itself, everywhere the state is displayed. And any state you have occupied longer than a few multiples of its historical median is an incident, whether or not a single threshold has been crossed. "TIER_3" reads as a status. "TIER_3 for 35d 4h" reads as a problem, and it reads that way to anyone, including someone who has never seen the system before.

What I Am Actually Doing About It

In order, cheapest and highest-value first.

A twelve-dollar thermometer on the motor housing. A DS18B20 strapped to the pump, reporting the temperature of the thing I actually care about. Everything above is downstream of not having one, and I have been running a 500-watt motor on inferred thermal data for five months to save twelve dollars and an hour.

Print time-in-state today, fix the ladder this week. The real repair is a de-escalation path with a trigger of its own. The five-minute version ships first, one line in the daily digest: STATE: TIER_3 for 35d 4h (4,209 cycles), last transition 2026-07-21 23:14. A digest that says "35d" next to "healthy" gets acted on. One that says OVERALL: OK never will.

Split the two thresholds. The abort check and the warning alarm currently share one constant, and they are answering completely different questions. One is "is this worth telling a human about." The other is "is it unsafe to actuate right now." Separating them is a one-line change and it is the single highest-leverage fix on this list, because it addresses the 44 percent directly.

Re-baseline the alarm against measured idle, so it cannot silently expire again the next time my duty cycle changes.

One Question for Anyone Buying This

Yesterday's question for vendors was: show me the success rate of every automated action your system takes. Here is this month's second one, and it is shorter.

Ask for the list, split into those two columns. Any monitoring product built on off-the-shelf smart plugs, relays, or controllers will have readings in the second column, and that is fine, they are useful readings. What you are testing is whether the vendor knows which column each one is in. If they cannot draw the line, then somewhere in their product a measurement of their own equipment is being presented to you as a measurement of yours, and it will drift with their behavior rather than with your machine.

Meanwhile, in the Basement

The pump ran again at 07:09 this morning. Cycle 4,209 out of a counter that reset to 1 on the twenty-first of July, drawing 501.2 watts at 48.6 degrees on 123.2 volts. The plug reports 851 hours of uptime and has not rebooted once since the day it was installed.

That is the thing I keep coming back to. A fifteen-dollar plug and a hundred lines of control logic have now run flawlessly through seven consecutive failures of the software I wrapped around them. Every single failure has been in my layer, and not one of them would have been prevented by a better model, more compute, or anything anyone is currently selling as AI. The hardware has been honest the entire time. It told me its own temperature every thirty seconds for five months, accurately, and I was the one who decided that number described the pump.

The repair routine will run again around 09:00. It will be attempt 754. Based on the last 519 and the current idle reading of 48.6 degrees, it will either fail or be cancelled by the temperature of the relay attempting it, and it will report whichever one it is to a channel with nobody on the other end.

The flashlight is still on my list. Thirty-five days in, it retains a perfect record, and it remains the only instrument in this entire story that I have not used.