My Pump Has Two Weather Systems and One Is a Constant.

Every status email my sump pump monitor sends contains this line:

Weather risk: 0.30

It has read 0.30 in every single one since June 2nd. Not because the weather has been remarkably consistent in upstate New York this summer. Because nothing has ever measured it.

0.30 is the value the state object is constructed with. Line 149 of pump_state.py:

self.cached_weather_risk = 0.3
self.last_weather_check = 0.0

That field is written in exactly one other place in the entire codebase, a function called fetch_weather_risk(). Its first two lines are:

if not AI_ENABLED:
    return

And the .env file on the monitor host says AI_ENABLED=false. It has said that since the day the service was deployed.

It is not a decorative number

If cached_weather_risk only appeared in an email I would have filed this under cosmetics. It doesn't. It feeds a function called weather_multiplier(), which feeds adjusted_rest(), which sets the actual OFF phase of the pump's duty cycle — how long the relay stays open between pulses when the system is managing a pump it believes is stuck.

Here are the bands:

0.3 is not ≥ 0.5, and it is not < 0.2. It falls into otherwise. For 114 days the weather-adaptive rest period has computed rest_s × 1.0 and returned exactly the number it started with. It is an identity function wearing an adaptive component's name.

What I checked, because "I think it's off" is not a finding

I have wasted whole days on this system by declaring something broken on the strength of one stale file, so the rule now is that any claim about a channel needs the raw evidence for that channel in front of me.

The configured URL is worth a sentence of its own. It points at a hardcoded 192.168.68.67. My Mac, which is where that analyzer was supposed to run, is on .149 today. Even if I flipped the flag to true this afternoon, it would connect to nothing — the identical hardcoded-IP drift that orphaned my on-device failsafe and sat undiscovered for four months.

There is one piece of outside corroboration. Because of a separate bug — a blank recipients variable that has silently dropped 3,374 emails — only four status digests ever actually left the host, all on June 2nd. Three of them have bodies. All three say Weather risk: 0.30. Those are timestamped copies, sitting in Gmail, outside my control, of a number that was already a fossil on the day it was sent.

The other weather system works fine

There are two, and they have nothing to do with each other.

The second one is a 114-line module that calls Open-Meteo directly and answers one question: is it wet? Rain falling now, or at least 1 mm in the last 24 hours, or 2 mm forecast for the next 24. Three decisions depend on the answer:

1. The run-length cap. Three minutes dry, six minutes wet, before the monitor cuts power and starts a recovery cycle.
2. The escalation route. After a failed power cycle: wet climbs the tier staircase from the bottom, dry jumps straight to the top tier.
3. Alert urgency. A long-run cutoff is a log entry on a dry day and an urgent page on a wet one.

I went back through six months of logs and pulled every occasion where the verdict was written down — 37 of them — and checked each date against Open-Meteo's own historical archive for my coordinates.

Thirty-seven for thirty-seven. It gets the awkward ones right too — June 23rd and September 14th were both nearly rainless days following a 13 to 16 mm soaking, and it called them wet, which is correct, because that is when the water is still arriving in the pit.

I have written about this module before, but about a different problem: my two processes ask it the same question and draw opposite conclusions, so a rainy night produces a cutoff and an immediate reversal. That is still true, and it is not what this post is about. What I had not looked at was what happens when the module can't answer at all.

The field nobody reads

The function returns five fields. One of them is ok.

Nothing reads it. All six call sites across both processes read is_wet, and nothing else.

So consider what happens when Open-Meteo is slow, or my DNS hiccups, or the connection drops. The fetch swallows the exception, returns nothing, and the assessment comes back like this:

{"ok": False,
 "recent_rain_mm": None,
 "current_rain_mm_h": None,
 "forecast_24h_mm": None,
 "is_wet": False}   # safe default

Then it caches that for ten minutes.

Downstream, is_wet: False is a dry day. A ten-minute network failure and a cloudless afternoon are literally the same event to every consumer in this system. And the comment in the module says exactly why the author — me — picked that default:

Failures degrade gracefully — return is_wet=False so we don't spam URGENT mails when the weather API is down.

That is a completely sensible decision about email volume. I was thinking about my inbox. The problem is that the same boolean is wired into two things that move a relay, and neither was part of that reasoning. When that fetch fails:

• the run cap halves, from six minutes to three, so a pump legitimately running long during a flood gets cut off sooner because an HTTP request timed out;
• escalation skips two tiers and lands in the top one, the state this system already spends about 90% of its life in.

Both of those move in the wrong direction under exactly the conditions where it matters, because storms and network trouble are not independent events at my house.

I can't tell you if it has ever happened

This is the part I find genuinely uncomfortable. There is no record.

The guardian process has a line that logs weather check failed. It has fired zero times in 82,017 log lines. That sounds reassuring until you notice it is unreachable for the actual failure mode: the module catches its own exceptions and returns a dictionary instead of raising, so the caller's except block can never run. It would only catch something exotic, like the import itself breaking.

The only tell a failure leaves is in the body of an alert email, which would print recent=None mm, forecast_24h=None mm where the numbers belong. The guardian logs alert subjects, not bodies. And the whole system has consulted the weather at a recorded decision point 37 times in 114 days, out of something north of 15,000 pump pulses. The rest of the time the verdict is fetched, used, and thrown away.

Zero mentions of the word "weather" in six months and 10.9 megabytes of log, for a variable that decides how long my pump is allowed to run.

Two failures, one shape

The industry framing this month is that vision and robotics pilots don't die of model error. The Robot Report ran a field piece on outdoor robots last week naming water, heat, connectors and connectivity as the killers. This week the Edge AI and Vision Alliance published a talk titled What Kills Industrial Vision AI Pilots? (Hint: It's Not Model Performance). When two outlets put the same thesis in a headline in one week, it is no longer a contrarian take.

I agree with it, and I would push it one step further, because in my own system it wasn't water or heat either. It was a false in a configuration file, a default that resolves to "multiply by one," and a status field that has been reporting a constant as a measurement since June. There was no model. There was nothing to be inaccurate.

What the two halves have in common is that both of them fail into silence. The dead pipeline reports a plausible number instead of an error. The live one reports "dry" instead of "I couldn't reach the internet." Neither writes a line anywhere. A system that fails loudly gets fixed in an afternoon; these have been running since June.

What I am changing

1. Make "unknown" a third value, not a synonym for one of the answers. Every call site reads ok and branches on it explicitly. When the weather is unknown I will keep the permissive six-minute run cap and the gentle escalation path, and downgrade the alert — fail toward not cutting the pump, since an unnecessary cutoff during a storm is the expensive mistake and an extra three minutes of runtime is not.
2. Log the input to every decision, not just the decision. One line per consultation, including ok. Six months of logs contain no trace of a variable that controls a relay, and that is the actual defect here. Everything else is recoverable if the record exists.
3. Either start the analyzer or delete the adaptive path. A disabled adaptive component that silently resolves to 1.0 is worse than no adaptive component, because it publishes a number and invites you to believe it. If it stays, the hardcoded IP goes.
4. Stop printing a value without its age. Weather risk: 0.30 should be Weather risk: 0.30 (never updated), or it should not be in the email.
5. Add a liveness check to my weekly review. For every computed field in the status emails, count distinct values over the last 30 days. Anything with a count of one gets flagged. That single query would have caught this in June. It would also have caught the three other fields I found earlier this month that had exactly one value across three weeks. It is the cheapest thing on this list and the only one that generalizes.

That last one is the takeaway I would hand to anyone running equipment monitoring. You probably have a tunable in your system — a threshold that adapts, a score that weights, a setpoint that responds to conditions. Go and find out how many distinct values it has actually taken in the last month. If the answer is one, it is not adapting. It is a constant with a good story attached, and the story is the dangerous part, because you have been reading it in a status email and quietly believing something was being handled.