My Escalation Ladder Was Two Hours Tall. The Failure Is Twenty-Eight Days Old.

Yesterday I wrote about the float switch that stuck in my sump pit in late July, and about the monitoring system that has been running the pump on a timer ever since. I wrote it from the last six alert emails, because those were the ones sitting at the top of the inbox.

Today I went back and pulled the first forty.

They contain the diagnosis. They also contain two things that show yesterday's post was partly wrong. I would rather publish that than leave it standing.

The Night It Started

Here is what actually happened, reconstructed from the alert stream itself rather than from memory or from the newest message.

The system was not confused. It was not slow. It went from the first sign of trouble to the top of its own severity scale in two hours and four minutes, correctly, on a mechanical failure its own control loop had no way to detect.

Then it stayed there for twenty-eight days and ten hours.

Exactly One Email Asked for a Person

At 02:38 on July 22, thirty-seven minutes into the event, the system sent a message that ended with this line:

Please manually investigate when possible.

That is the only time in twenty-eight days that anything asked for a human. It came from TIER_2, a state the system left thirty-six minutes later, and the request left with it. It was never repeated, because the code that sends it belongs to a state the system was no longer in.

Everything after that says the opposite. The escalation notice says “This is designed escalation behavior, not urgent.” The 3,379 messages that followed say the float is still stuck, report the wattage and the temperature, and offer a manual override command. None of them ask for anything.

So the honest accounting is not that my system failed to alert me. It alerted me 3,379 times. It is that the one message that asked for a decision was tied to a state the incident passed through in half an hour, and the state the incident actually lived in was the one I had named “steady state.”

Where I Got It Wrong Yesterday

Yesterday I wrote this, and it is still live on this site:

The power readings for the last few days have been sitting at the low end of normal while the plug temperature sits about ten degrees above it, and for a centrifugal pump, lower power means less water moving.

I used that to argue the pump might be running dry. The power half of the argument still stands and is still unresolved. The temperature half does not, and the first forty emails are why.

At 01:15 on July 22, the system sent a temperature warning reading 50.3°C while the pump was drawing 0.0 watts. Zero. The plug was that warm with nothing plugged into it doing any work at all.

Which means the temperature I was treating as evidence of a laboring pump is mostly the plug warming itself in a July basement. My warning threshold is set at 50.0°C, so these alerts have been firing every time the ambient environment drifts across a line it crosses on its own. Today's readings are 49.2 to 49.7°C, which is to say: just under it, for reasons that have nothing to do with the pump.

I built a two-signal argument out of one real signal and one thermometer measuring the room. I would not have caught that from the top of the stack, because at the top of the stack the pump is always running when the temperature is reported. You only see the 0.0-watt reading in the first hour, when the plug was briefly off.

The Bug I Found on the Way

While reading the escalation code, I found a defect that did not cause any of this but will cause something eventually.

My system has a weather-aware branch. On wet days a stuck float might mean real water, so it steps gently up the ladder. On dry days, a stuck-closed float just runs the pump against an empty pit, so the code jumps straight to what the comment calls “TIER_3’s longest rest period… to minimize dry-pump wear.” That sentence is also in the email the system sends you when it happens.

Here is the actual configuration:

TIER_3 does not have the longest rest. It has the shortest rest of the three, and it runs the pump at 2.67 times the duty cycle of TIER_1. The branch that exists specifically to reduce dry-running wear jumps to the setting that runs the pump the most, and then emails you a sentence claiming it did the opposite.

It did not fire in my case. July 22 registered as wet, so I took the normal staircase. It is sitting there waiting for the next dry-day failure, and in the meantime the false explanation is being sent to a real inbox as reassurance.

What This Week Said About the Same Problem

Two things happened elsewhere in physical AI this week that are the same story at different scales.

FORT Robotics announced it is going public at a pro-forma enterprise value of $556.6 million. Six hundred customers, nineteen thousand five hundred deployed units, sixty-six percent gross margins. What do they sell? Wireless emergency stops and safety controllers, certified to SIL 3. Their stated mission is to ensure robots cause no harm.

Read that as a product category and it is striking. The market just capitalized, at half a billion dollars, a certified, independent way to halt a machine that its own controller cannot be trusted to halt. Not better perception. Not better planning. Interruption, with a certification number on it, living outside the system being interrupted. And sixty-eight percent of their 2025 bookings came from customers who signed before 2025, which means people keep paying for it after the pilot ends.

The second thing was a building-industry newsletter about what its authors call the proof tax: the cost energy managers pay proving their work was worth doing. One number in it stopped me. An energy manager at Texas Tech identified $977,000 in savings in his program's first year, and had the capacity and permission to capture $97,000 of it. A ten-to-one gap between what he proved and what he was allowed to do.

Another manager in that piece, explaining why he cannot get meters installed in buildings he knows are wasting energy, said something I have not been able to put down:

The issues don't seem to ‘impact’ anyone, so they don't get prioritized.

The Part That Is Actually About My Product

That sentence is my basement, word for word.

My basement is dry. The pump runs. Nobody is inconvenienced. And that is precisely why twenty-eight days went by. My system was good enough at compensating for the failure that it removed the impact, and the impact was the only thing that was ever going to make anyone act.

I sell that system.

And there is one more thing I have to say plainly, because it is on the page you would land on if you clicked through to hire me. My physical AI page has four pillars, and the third one reads:

AI Advises, Never Controls. Safety-critical systems need human oversight.

My reference implementation has been a pump's controller for twenty-eight days with zero human oversight. The manual override exists. It is printed in every single one of the 3,379 emails. Nobody used it, and nothing in the system required anyone to.

That claim is not a lie. It is unowned. “Human oversight” written on a sales page is not a mechanism. An override that nobody is obligated to use is not oversight. It is a button in a room where no one is standing.

Three Rules I Am Paying For

An escalation scale must span the duration of the incident, not the duration of the diagnosis. Mine was sized for how long it takes to figure out what is wrong, which is minutes. It should have been sized for how long the thing can stay wrong, which is months. Every tier I built is about severity. Not one is about time.

A takeover needs a clock, an owner, and a hard stop. Taking over a failed control loop is legitimate as a bridge. Mine has no expiry, no named human, and no terminal state, which is how a bridge becomes a residence. This is the product FORT just took public at half a billion dollars, in the industrial version: an independent thing whose only job is to stop the machine and hand it to a person.

Read the alert stream from the bottom. Every dashboard, every inbox, every incident view I have ever built or used sorts newest first. In a twenty-eight-day incident, the newest message is the least informative one in the entire archive. The power event, the plug that came back with its output off, the 50.3°C reading at zero watts — all of it was available on July 22 and none of it is in the message that arrived twelve minutes ago. Nothing in my product ever shows anyone the first hour again. That is a feature I am now going to build.

What I Am Doing About It

Going to look at the pit, which I said yesterday and have still not done. The cycle counter advanced exactly 120 overnight, which is precisely what twenty-four hours at twelve minutes a cycle produces with no interruption. That number is proof that nobody touched anything.

Shipping the duration tier. One comparison against a timestamp I already track, one new terminal state. Twenty-four hours in TIER_3 stops the heartbeat and sends one message, to a different address, with a subject line that does not contain the words “steady state.”

Fixing the backwards rest-period text before the next dry-day failure fires that branch.

And changing the third pillar on my own sales page. Not to weaken it — to make it true. The claim I actually believe, and can defend, is bounded takeover: the system may drive the machine to protect it, for a defined window, after which it must stop and produce a human. That is a better product than “never controls,” and unlike the current wording, my system will actually do it once the duration tier ships.

One hundred and twelve hours of motor runtime. Fifty-four kilowatt-hours. Three thousand three hundred and seventy-nine proofs. I was never short on evidence. I was short on anything that turned evidence into a decision with a name and a deadline on it, and I have been selling the evidence.