A Ten-Dollar Float Failed. My Software Has Run the Pump 3,259 Times Since.

This morning, my sump pump monitoring system sent an email. It said the float switch was still stuck, that the pump was drawing 488.7 watts, that the plug was at 48.5°C, and that this was cycle 3,259.

It sent a nearly identical email twelve minutes earlier. And twelve minutes before that. It has been doing this, without a gap, since roughly the twenty-second of July.

I did not know that until this morning. I found it while researching something else entirely.

What Actually Happened in the Basement

A sump pump does not have a brain. It has a float — a mechanical switch, a piece of plastic on a hinge, worth about ten dollars. Water rises, the float rises, the switch closes, the pump runs. Water falls, the float falls, the pump stops. That is the entire control system, and it has been the entire control system for about seventy years.

Sometime in late July, mine stuck. Not the pump. The float. Which means the pump lost the only thing that was ever going to tell it to stop.

My monitoring system noticed. It could not see the water and it could not see the float — it has no sensor for either. What it could see was that the electrical signature had gone wrong: the pump was running when the pattern said it should have stopped. So it escalated, the way I built it to. It tried a power cycle. It tried a series of increasingly aggressive rapid-cycling routines designed to physically shake the float loose. None of them worked.

Then it did the thing I am genuinely glad it did. It took the pump off the float entirely and started running it on a timer: two minutes on, ten minutes off, indefinitely. Enough to keep the pit down. Not so much that the motor cooks.

That was twenty-seven days ago. In that time it has run the pump about 3,259 times, for roughly 109 hours of accumulated motor runtime, and sent me about 3,259 emails saying so.

So Here Is the Part That Bothers Me

Go back and read that sentence again: it sent me about 3,259 emails.

An alert that has repeated 3,259 times is not an alert. It is a new normal that nobody has to sign for.

I built this system. I wrote the escalation logic. I am the most motivated possible operator of it — I write publicly about it, I sell services based on it, my own house is the thing at risk. And it produced three thousand notifications about an active mechanical failure, and I did not act on a single one of them.

Not because I ignored them. Because after the first day, they stopped being information. Twelve-minute intervals turn an alarm into a heartbeat. By week two, an email that says the system is handling it is indistinguishable from an email that says the system is fine.

And the code agrees with that reading. Here is the actual comment in my source file describing the state the pump has been in for a month:

TIER_3 - 120s ON / 10 min OFF. Max duty, alerts every cycle.

The email subject line my system generates for it is worse. It reads: “TIER 3 cycle 3259 (steady state max duty).”

I called it steady state. I named the failure mode after the word for “nothing is changing,” and then I built a system that is perfectly content to remain there forever, and then I was surprised that nothing changed.

The Design Bug Is Not Where I Would Have Guessed

If you had asked me last week what could go wrong with this system, I would have talked about sensor coverage, or threshold tuning, or what happens when the network drops. Real problems, all of which I have written about.

The actual bug is simpler and I did not have a name for it. My system has two states of the world: handled and unhandled. It escalates hard on severity — that is what all those tiers are — and severity has been pinned at the ceiling since July. But it escalates not at all on duration. There is no path out of TIER_3 except success. Nothing in the design asks the question a human would ask on day three:

How long has this been “handled”?

Twenty-seven days of compensation and twenty-seven minutes of compensation produce exactly the same behavior from my software. The only difference is a counter in the subject line that nobody reads, which is a fantastic description of how a number becomes invisible.

So here is the rule I paid twenty-seven days to learn, and I think it generalizes well past sump pumps:

A Post of Mine This Contradicts, and How I Am Resolving It

I wrote a piece a while back called Watch the Pump, Don't Drive It. The argument was that a monitoring system should observe the asset, raise a hand, and let the existing controls do the controlling. I still believe the general shape of that.

My system is currently driving the pump. It has been for a month. There is no way to read that except as my own software doing the thing I publicly argued against.

Having now lived on the wrong side of it, I do not think the takeover was the mistake. When the existing control has failed — not degraded, failed, as a stuck float has — there is nothing left to defer to, and refusing to act on principle would have meant a flooded basement. The takeover was correct.

The mistake is that I built a bridge and never built the far end. The revised rule I actually hold now: don't drive the pump — unless the driver is gone, and then drive it on a clock that runs out and forces somebody to look. My system got the first half right and did not know there was a second half.

What This Should Mean If You Own a Building

Everything above is one pump in one basement. But the failure is structural, not personal, and it scales in an unpleasant direction: the more reliable your monitoring is at compensating, the less likely anyone is to notice that it is compensating.

Two questions worth asking any monitoring vendor, including me:

• “What happens when your system handles a problem instead of fixing it? Who finds out, and when?” If the answer is “it alerts,” ask how often, and to whom, and what changes on day ten. An alert that fires every twelve minutes for a month is functionally the same as no alert, and it will show up in the logs as diligence.
• “Does anything in the system escalate on duration alone?” Most monitoring escalates on severity. Very little of it escalates on how long a condition has persisted. Those are different failure modes and the second one is the one that quietly eats equipment.

There is a real cost hiding in that second question. My pump has now run 109 hours it was never designed to run, on a duty cycle chosen by a constant I typed, protecting against a thermal limit I am inferring from a plug rather than measuring at the motor. Every one of those hours was a small withdrawal from the life of a machine, made silently, with my full technical awareness and zero attention.

What I Am Doing About It This Week

First, and before I write another line of code: I am going down there with a flashlight. The power readings for the last few days have been sitting at the low end of normal while the plug temperature sits about ten degrees above it, and for a centrifugal pump, lower power means less water moving. That combination is what a pump cycling against an empty or near-empty pit looks like. I have no water-level sensor, so I cannot resolve that from my desk at any amount of cleverness, which is its own lesson about instrumentation.

Second, the code change, and it is small. One comparison against a timestamp I already track, and one new terminal state. If TIER_3 persists past some number of hours, stop sending the twelve-minute heartbeat and send one message, to a different address, with a subject line that does not contain the words “steady state.” A reader asked me for two-stage alerting six days ago and I have been carrying it as a nice-to-have. It is not a nice-to-have. My own system just spent a month explaining why.

Third, I am going to price a water-level sensor. Twenty to forty dollars for the one instrument that would have collapsed all of this ambiguity on day one. I have been reasoning about a pit full of water by looking at a wattmeter for two years, because the wattmeter was already there.

The uncomfortable version of this whole story is that I have spent the last several weeks writing thoughtful posts about sensor selection, threshold derivation, and system extensibility, and during every one of them my own pump was running on a stuck float and telling me so every twelve minutes. I found it this morning by accident. The monitoring was never the thing that failed. The reading of it was.