Twenty-Two Days of HIGH_INFLOW From a Pump on a Timer

My sump pump monitor mails me a status digest every morning at eight. Buried in it is a line where the system says what it thinks is going on in my basement. Yesterday's read:

HIGH_INFLOW INFO | Pumping a lot but pump is healthy (12 min/hr)

I could not get to the monitor host this morning. My laptop was on a different network than the pump, so no log, no shell, nothing. So I did the thing I should have done a month ago and pulled the last twenty-two of those digests out of my mail instead, and lined the assessments up next to each other.

All twenty-two say HIGH_INFLOW.

Not a wet week. Not a storm. Every day the assessor has run, going back three weeks, it has reached the same verdict. And every day it has filed that verdict as INFO, and every day the header of the same email has said OVERALL: OK.

The Table

Two fields from each digest: the runtime the assessor measured, and what it concluded.

Range is 8 to 17 minutes per hour. The low end, 8, shows up on thirteen of the twenty-two days. Which tells you something the source code would have told you faster: whatever the HIGH_INFLOW threshold is, it is at or below 8 minutes of pump runtime per hour.

Hold that number.

What the Controller Is Doing at the Same Time

Every one of those digests also says state=TIER_3.

TIER_3 is the mode my monitor drops into when it decides the float switch is stuck. The float is the ten-dollar part that is supposed to tell the pump when there is water. Mine has been stuck since the start of September, and it has been stuck on and off since summer. When TIER_3 engages, the monitor stops trusting the float and starts running the pump on a fixed timer through the smart plug: 120 seconds on, 600 seconds off. Five cycles an hour, two minutes each.

That is 10 minutes per hour. Commanded. By me. By my own software, on a clock, regardless of whether there is a drop of water in the pit.

The assessor is reading the pump's runtime and inferring groundwater. That inference is sound — right up until something other than the water starts deciding when the pump runs. Then the arrow flips. Runtime stopped being caused by inflow the moment my controller began setting it, and every conclusion downstream of runtime went with it.

For twenty-two days my system has been measuring its own timer and reporting the result to me as a fact about my basement.

Why the Threshold Was Right When It Was Written

I want to be fair to the code, because the mistake here is more interesting than carelessness.

When I wrote that threshold, the pump ran on its float. Water rises, float lifts, pump runs. Under those conditions runtime per hour is a genuinely good inflow proxy — cheap, requires no extra sensor, and 8 min/hr really did mean the water table was up. The threshold was correct, and it was correct for a real reason.

Then I added TIER_3. TIER_3 is a good feature; it is the reason a stuck float has not flooded my basement. But it quietly invalidated a threshold in a different file, written months earlier, that had no idea it existed. Nothing errored. Nothing crashed. The classifier kept computing, kept returning a value, kept formatting it into an email. The output stayed the same shape. Only the meaning fell out.

This is the failure mode I would now put at the top of the list for any monitoring system that can also actuate something:

The moment your software can act on the equipment, your readings stop being independent observations of the world. They become partly a record of your own decisions.

Where Else This Hides

Once you have the shape, it is everywhere in building automation, and it always arrives the same way — a control feature added later, breaking an inference written earlier.

• A demand-response controller cycles a rooftop unit to shave peak load. The runtime-based efficiency alert now grades the demand-response schedule.
• A setback schedule pre-heats a space at 5 a.m. The occupancy model, which infers people from thermal activity, now sees a building that comes to life before anyone arrives.
• A lighting controller sweeps floors on a timer. The after-hours-usage report is now a report about the sweep.
• A monitor runs a pump on a timer. The inflow classifier grades the timer. Mine.

The tell is always the same, and it is embarrassingly easy to check: a field that never changes is not telling you anything, no matter what it says. Twenty-two identical classifications in a row is not a three-week weather event. It is a constant that has been dressed up as an observation, and it had been sitting in my inbox in plain sight the entire time.

The Ten-Minute Audit

You do not need the source code to run this, which is the useful part. Take any recurring status message you get — from your own system or from a vendor's — pull the last thirty, and count the distinct values in each field.

Any field with exactly one distinct value across a month is a suspect. It is either broken, or it is decorative, and both of those are worth knowing. In my digest, three fields failed: OVERALL (22 out of 22 OK), the assessment class (22 out of 22 HIGH_INFLOW), and the severity (22 out of 22 INFO).

Three of the fields in my daily health report carry no information. I have been reading that email for three weeks with the vague comfort of a person who is being kept informed.

The Fix Is Not a Better Threshold

My first instinct was to raise the number — move HIGH_INFLOW above 10 min/hr so the timer stops tripping it. That is wrong, and it is wrong in a way worth naming, because it is the instinct almost everyone has.

Raising the threshold produces a classifier that is quiet during TIER_3 and still meaningless during TIER_3. It would stop saying the wrong thing and start saying nothing, while continuing to look like it was working. That is worse, not better, because now the silence reads as good news.

The honest fix is for the assessment to know when its own input has been compromised and say so:

INFLOW: UNKNOWN — runtime is controller-set (TIER_3 active)

That is a worse-looking status line and a far better one. It tells me the truth, which is that while the timer is driving the pump I have no inflow measurement at all, and if I want one I need a sensor that measures water instead of a sensor that measures my own software. A float position, a pit level, a moisture reading. Something with an independent view.

Generalized, the rule I am adopting: every threshold should name the actuator it is not allowed to be downstream of. When that actuator engages, the assessment suspends and announces the suspension. An honest gap beats a confident constant.

If You Are Buying Monitoring

I keep publishing my own system's failures because they map cleanly onto questions worth asking a vendor, and this one produces a good one.

Most monitoring pitches lead with detection: here is what we catch. A smaller number will talk about routing and delivery, which is where my last two posts found problems. Almost nobody will volunteer the question underneath all of it, which is whether the analysis is looking at the equipment or at the vendor's own control layer.

Question two takes ten minutes, needs no technical knowledge, and no demo survives it, because a demo is a good day and a good day is exactly when every field reads the same.

My honest answers today: yes, my system controls the pump, and yes, one of my classifications has been reading my own timer for three weeks. The change is a few lines and a restart on the machine in my basement, which I could not reach this morning — so it queues behind two other one-line fixes I have written about here and have not yet landed. That queue is now three deep, and every item in it is a single line. It is starting to look less like a backlog and more like a habit.

The pump, throughout all of this, is fine. It has run 806 cycles in the current stretch, pulled a healthy 480-odd watts on both of the mornings the digest happened to catch it mid-pulse, and kept my basement dry through a stuck float that would have flooded it weeks ago. The hardware has been doing its job perfectly. It is the paragraph describing the hardware that has been quietly wrong.