I Blamed My Code. The Log Said It Was August.

Yesterday I published a post about a bug in my sump pump monitor. A single temperature constant was doing two unrelated jobs: printing a chatty warning in the log, and, five hundred lines away, aborting the only self-repair mechanism the system has. One hundred and forty-nine repair attempts had quit halfway through over a temperature that meant nothing.

That part was true, and it is still true today.

But I also offered an explanation for why those aborts happened, and that part was wrong. I want to walk through how it was wrong, because the mistake is one I have watched other people make in front of paying customers, and the thing that caught it was sitting in the same file the whole time.

The Story I Told Myself

The unstick routine snaps a 490-watt pump on and off six times in about a minute. Halfway through, at cycle three of six, it reads the relay temperature and bails out if it is above 50 °C.

So here is a tidy little tragedy: the routine switches a heavy inductive load through a small relay, the relay warms up from the switching, the routine then measures that warmth and cancels itself. The repair defeats itself by attempting the repair. The harder it tries, the more certain it is to quit.

I liked this explanation a great deal. It is mechanical, it is specific, and it makes the code the villain, which is satisfying when the code is yours and you are about to fix it.

I even had evidence. I bucketed all 149 aborts by hour of day, expecting to see them clustered in the afternoon if the cause were an ordinary hot basement. They were not clustered at all. Eleven aborts at two in the morning. Flat across the clock. Ambient heat has a daily rhythm; these did not. Therefore, I reasoned, not ambient. Therefore self-inflicted.

I hedged in the post, said I could not fully isolate the mechanism without a temperature reading taken before the sequence starts, and that the code does not take one. Then I published.

The Reading I Said I Did Not Have

I did have it. I had 14,352 of them.

The unstick routine takes a relay temperature at its midpoint. But the monitor also logs a relay temperature on every ordinary pump cycle, all day, every day, when the unstick routine is not running at all. Every one of those lines looks like this:

[2026-09-06 07:03:59] TIER_3 cycle 445: pump ran (486.8W, 48.9C)

That is a control group. It is a measurement of exactly the thing I needed — what this plug does thermally on its own, without my repair routine touching it — and it has been accumulating since February. I had been reading these lines for weeks. I had been pulling the wattage out of them for a different analysis. I had never once pulled the temperature out and asked what the distribution looked like.

Here is what it looks like:

ORDINARY CYCLE-TIME RELAY TEMPERATURE, ALL TIME
n = 14,352    min 24.7 C    mean 46.61 C    max 52.7 C

  > 49.0 C :  5,533  (38.55%)
  > 50.0 C :  2,302  (16.04%)   <-- the abort gate
  > 50.5 C :    718  ( 5.00%)
  > 51.0 C :     88  ( 0.61%)

The plug sits above my abort threshold 16% of the time on its own. No repair routine involved. No rapid switching. That is just what this device does in this basement.

And the abort rate for the unstick routine? 18.2%.

A rate means nothing until you know the base rate. Eighteen percent sounds like a defect. Against a sixteen percent background, it is barely an observation.

Splitting It By Weather

The all-time comparison is suggestive, but relay temperature swings enormously across the year, so I did the version that actually settles it. For every day in the log, I took that day's ordinary base rate above 50 °C and used it to predict how many unstick attempts should have aborted. Then I compared to what happened.

Ninety-two percent versus one half of one percent.

The day-by-day view is even blunter. On August 9, the plug was over 50 °C on 120 of 120 ordinary cycles, and 6 of 6 unstick attempts aborted. On August 16, it was over 50 °C on 0 of 119 cycles, and 0 of 6 aborted. The abort series tracks the ambient series almost line for line, for seven months.

Across the whole log, the same-day ambient base rate predicts 115.7 aborts. I observed 149. So there is a real excess — about 1.3× — and it points the direction I originally guessed. The routine does appear to add something on the order of a few tenths of a degree by its midpoint. But that is a modifier on the edges. Roughly 78% of the aborts are explained by the basement being hot, and everything my code does accounts for the rest at most.

My repair routine is not sabotaging itself. It is August.

What Went Wrong In My Head

The hour-of-day analysis was not fabricated. I ran it, and it showed what I said it showed: aborts were flat across the clock, with no afternoon peak.

The error was in what I concluded from that. Aborts were flat across the clock because they were concentrated in a two-week stretch in August when the basement was above 50 °C at every hour of the day and night. There was no daily rhythm to find, because during that window there was no cool part of the day. I checked for an hour effect, correctly found none, and then read "no hour effect" as "not ambient."

That does not follow. I had stratified on a variable that happened to be easy to compute, found nothing, and treated absence of one confounder as evidence for my preferred mechanism. The variable that mattered was the date, and I never looked at it, because looking at it required admitting the boring answer might be right.

The Fix Is The Same. The Reason Is Worse.

None of this rescues the threshold. It is still a warning constant moonlighting as a safety interlock, still set 10 °C below the cutoff that actually protects anything, still cancelling repairs for no reason. The fix I proposed yesterday — split the constant, raise the abort gate to 55 °C, sample it once before the sequence instead of in the middle — is exactly the right fix.

In fact today's data justifies the specific number better than yesterday's did. Across 14,352 readings spanning seven months and a full summer, this plug has never exceeded 52.7 °C. A gate at 55 sits above everything the device has ever actually done, and still 5 °C below the soft cutoff and 15 °C below the point where the electronics quit.

But the character of the bug changed completely, and it changed for the worse.

If the aborts had been self-inflicted, this would be a structural flaw firing at a steady rate all year. Annoying, findable, and honest — the kind of thing that shows up in any test.

Because the aborts are ambient, it is a seasonal failure. And a seasonal failure has a property that a constant one does not:

• From October through June, the self-abort rate is 0.5%. The system looks fine. It is fine.
• For two weeks in August, it is 92%. The only repair mechanism in the system is effectively offline.
• Those two weeks are peak summer storm season — the exact window when a sump pump is doing the work you bought it for.
• Nothing in the alerting would ever mention this. Not one URGENT notification has fired in 46 days.

So the recovery mechanism is least available precisely when it matters most, and it is silent about it.

This Is Why Pilots Lie

Here is the part I would put in front of anyone buying monitoring, for a building or a plant or a rental portfolio.

Suppose I had built this for a customer and run a proper 90-day pilot starting in February. Ninety days of clean data. The unstick routine would have shown a 0.5% self-abort rate. I would have written it up, they would have signed, and the system would have worked exactly as advertised straight through spring.

Then August would have arrived, the abort rate would have gone to 92%, the basement would have flooded during a thunderstorm, and every log line and every dashboard would have said the system was operating normally — because by its own definitions, it was.

Any acceptance test that runs in one season has tested one season. A system that passes in March has told you what it does in March.

I do not think most people selling building monitoring have run this check on their own products. I had not run it on mine, and I wrote mine, and I have been reading its logs daily for six months.

If you are evaluating a monitoring vendor, the question worth asking is not what the system detects. It is: what does your detection logic do at the temperature extremes of my building, and what data do you have on that rather than what do you believe? If the answer involves the word "should," you have your answer.

Your Alarm History Is A Dataset Nobody Has Queried

The practical version of all this, and the reason I keep writing these up: almost every building automation system on earth is recording two things side by side. A continuous trend log of raw sensor values, and a much shorter record of alarms — the moments a value crossed a line.

Everyone looks at the alarm record. It is short and it is exciting. Practically nobody queries the trend log to find out how often each alarm's underlying condition occurs in general, including all the times it crossed the line and nobody happened to be looking.

That comparison sorts your alarms into two piles:

• Alarms that fire at the background rate. These are measuring the weather. They tell you it is August. They are the ones your staff learned to ignore in the first month, and they were right to.
• Alarms that fire above the background rate. These are measuring your equipment. Something is actually degrading. These are the ones nobody can find, because they are buried in the first pile.

Separating those two piles is roughly a week of work. It needs no new hardware, no new sensors, and no site visit. It needs somebody to read what the building has already been writing down for years and ask a question the alarm architecture structurally cannot ask, because an alarm system only records the crossings and never the denominator.

I ran that query on my own basement this week and it told me I was wrong about my own code, which I had written, about a bug I had found myself, and had already published. That is a fairly strong argument for running it on a building where nobody has looked at all.

Where The Pump Actually Is

Still stuck, for whatever it is worth. Ninety hours in the top escalation tier now, 445 cycles, every single one drawing full power, the float unmoved. Roughly 22 unstick attempts since the lock began, and none of them aborted — September base rate, exactly as the model now predicts. They all ran to completion and failed honestly.

Fixing the threshold would not have saved this particular lock. Sometimes the software really is out of moves and the correct next step is a flashlight and a basement, which is where I am headed after I publish this.

Meanwhile the stuck state is costing 2.014 kWh a day — four consecutive daily readings now, holding within half a percent, about $162 a year to run a pump that is accomplishing nothing.

Six days running, this system has corrected something I believed about it. Today it corrected something I had already published, using data it had been handing me since February. I would rather find that out from my own basement than from a customer's.