My Monitor Took 207 Readings and Learned Nothing

Over the last forty-two hours, the monitoring system in my basement took two hundred and seven measurements of my sump pump.

Here is the entire dataset, summarized honestly:

Every one of those readings is fine. A half-horsepower sump pump under load draws about five hundred watts. Five hundred watts is what a working pump looks like. There is no threshold anywhere in my system that this data violates, which is exactly why my system has said nothing about it for forty-four days.

The pump is stuck. It has been stuck, in the most serious state my monitor has, for the better part of two months. And the reason nobody was told is not that the measurements were wrong. The measurements were excellent. The measurements were the problem.

The Alarm Nobody Builds

Every alarm I have ever written fires when a number crosses a line. Too hot. Too cold. Too much current. Ran too long. That is what alarms are, and if you have a building management system, essentially everything in it works this way.

None of them fire when a number stops moving.

A reading that has not changed in two days is not a healthy reading. It is a reading that has stopped being an input.

Think about what a genuinely healthy piece of equipment looks like in data. It has texture. The load shifts as conditions change. Ambient temperature drifts across the day. Duty cycles vary with demand. A motor that runs harder in the afternoon than at four in the morning is a motor responding to the world, and that responsiveness is visible as variance.

Now think about what my pump looks like. Two hundred and seven readings agreeing with each other to within ten percent, across two days, day and night, through a rainstorm and through a dry afternoon. Flat. Glassy. Utterly consistent.

That is not a healthy signal. That is a signal where one variable has pinned every other variable, and the pinned variable is the failure. The float in my sump basin is stuck closed. Every time my monitor gives the pump power, the pump runs at full load, because as far as the pump is concerned it is always supposed to be running. The reading never varies because the condition never varies. Nothing else about the world is reaching the sensor anymore.

I had the data to see that in February. I looked at it for the first time this week.

The Door That Is Only Checked 120 Seconds Out of Every 720

It gets worse when you read the code, which I finally did instead of inferring behavior from the log.

My monitor escalates through tiers when the pump appears stuck. The top tier runs the pump for 120 seconds, rests for 600, and repeats. There is no tier above it. There are exactly two ways out:

Any tier → 3 consecutive dry pulses → NORMAL
Any tier → pump stops on its own → COOLDOWN → NORMAL

Both of those require the float to physically release. Neither of them is something my software can cause. That is a design flaw I already knew about.

Here is the one I did not know about. The dry-pulse counter is only ever evaluated during the 120 seconds the monitor is actively forcing the pump on. During the 600 seconds of rest — 83% of every cycle — the plug is off and no evidence is gathered at all.

So my system spends five sixths of its life deliberately blind, and the remaining sixth asking a question it has now asked two hundred and seven times and received the same answer to every single time. The counter it needs to reach three has been sitting at zero since Wednesday, because it resets on every wet pulse and every pulse has been wet.

I built a machine that generates about nine hundred log lines a day, consumes real electricity doing it, and has produced exactly zero bits of new information since Wednesday afternoon.

The Meter That Was Already Running

The part of this I am actually glad about is small and stupid and took ninety seconds.

The pump is plugged into a fifteen-dollar smart plug. That plug does one thing besides switching: it counts lifetime energy, because that is a standard feature on every metering plug ever made. It has been counting since February. I had never once read the counter on two consecutive days.

So I did.

Two kilowatt-hours a day, essentially all of it going into cycling a pump that cannot fix itself. That works out to about 732 kWh a year, or roughly $161 annually at what I pay for electricity in New York.

A hundred and sixty-one dollars is not going to bankrupt me. That is not why it matters. It matters because it is a number, and until this morning the failure did not have one. "The pump has been stuck for six weeks" is a sentence people nod at and forget. "The pump being stuck costs forty-four cents a day, every day, and has been doing so since July" is a sentence that gets a work order written.

The irony that I keep turning over is this: the same device that failed to tell me the system was stuck was, that entire time, precisely and continuously metering what being stuck cost. The alarm and the invoice were on the same chip. I was reading neither.

Three Things to Check on Your Own Systems

None of this requires new hardware, and all of it is a rolling window and a comparison.

1. Alert when variance collapses, not just when it spikes. Compute a rolling standard deviation on your key signals — a 24-hour window is a reasonable start — and raise a flag when it drops well below its own trailing median. Everyone alarms on a signal that jumps. Almost nobody alarms on a signal that goes flat, and flat is what a pinned variable looks like from the outside.
2. Audit your state machines for exits that require the fault to fix itself. Write down every terminal state. For each one, write down what physically has to happen to leave it. Then ask whether your software has any influence over that thing at all. Where the answer is no, that is not a state, it is a destination, and it needs a wall-clock escape hatch that pages a human after N hours no matter what the readings say.
3. Find the meter that is already running on your failure. Read a lifetime counter today. Read it again tomorrow. Subtract. You now have the daily cost of whatever is currently wrong, in the customer's own units, priced off equipment nobody has to buy.

What This Means If You Own a Building

Your building management system is very good at telling you when something crossed a line. It has no opinion at all about something that has been sitting in the same place for six weeks.

That is not a criticism of any particular vendor. It is a description of how alarm architecture works everywhere. Alarms are built around events, and an event is by definition a change. The industry has spent thirty years getting very good at noticing change and has built almost nothing that notices the absence of it.

Which is unfortunate, because that is precisely where the expensive failures live. Fast failures announce themselves — something trips, something floods, somebody calls you. Slow failures do not. A pump running at full load around the clock, a valve that has been ninety percent open since spring, a fan running against a dirty filter for a full season. All of those look completely normal to a threshold. All of them are quietly spending your money and your equipment's life, and the first time anyone finds out is when the equipment is finished.

Industry benchmarks put emergency repairs at three to five times the cost of planned work. That multiple is the entire argument for condition monitoring, and it is one you can check yourself against your own last two invoices without believing a word I say.

So the numbers I have started putting in my customer reports are not the ones I expected to be putting there a year ago. They are:

• Duration in state. How long has each piece of equipment been in a condition it was not designed to hold?
• Time to recurrence. When something got fixed, how long did the fix actually last before the same condition came back?
• Signal variance. Is this reading still responding to the world, or has it gone flat?
• Cost of the abnormal state. Priced off meters already installed in the building, in dollars per day.

I did not arrive at that list from market research. I arrived at it because I spent three days reading a 144,000-line log off my own basement and discovered that I needed all four of those numbers and had exactly none of them.

The Uncomfortable Part

I should say plainly that this is the third time this week my own equipment has corrected something I published about it.

On Tuesday I wrote that the escalation ladder had been welded shut for forty-two days and that only a power failure would reset it. On Wednesday it reset itself, unassisted, and I got to write about how the recovery lasted seven minutes. Today I found out that the exit condition I have been describing as "stuck" is one the system only checks during a two-minute window out of every twelve, and never at any other time.

Three days, three progressively less flattering readings of the same log file. Every one of those findings was available in February. None of them required a new sensor, a better model, or a single dollar of hardware. They required somebody to sit down with data that was already being collected and ask a question that was not a threshold.

That is most of what this work actually is. The sensors are cheap and getting cheaper — vibration monitoring that cost six hundred dollars a point three years ago is fifty dollars today. The measurement is a solved problem. What is not solved, and what nobody is selling to small buildings, is somebody asking your data a question that your alarm system structurally cannot ask.

My float is still stuck, by the way. I am going down to the basement after I publish this. The software has no moves left, which is a fine thing for the software to admit and a much better outcome than six more weeks of a machine confidently logging the same number.