My Crash Table Had Six Rows and Four Causes.

Two pieces ran this week arguing that field AI does not die of model error. The Robot Report published "Why outdoor robots fail", listing temperature, water ingress, connectors, battery and connectivity. The Edge AI and Vision Alliance posted a Kasqade talk titled "What Kills Industrial Vision AI Pilots? (Hint: It's Not Model Performance)." Both are right that the model is rarely the problem. Here is a smaller, duller failure that neither list includes, and that I shipped into my own published work three weeks ago.

The table I published

My monitor watches the smart plug that switches my sump pump. When the plug's uptime counter goes backwards, the monitor reads the device's reset_reason integer, looks it up in a dictionary, and writes the human-readable name to the log. Counting those names across seven months gave me this:

Six reset reasons. Reasonable-looking. A main failure mode and a tail of rarer ones, which is what failure distributions usually look like, which is exactly why I didn't examine it.

What the numbers actually say

Count the same events by the raw integer instead of the decoded string and the table collapses:

Code 4 accounts for 205 of 231 reboots. "OWDT_RESET" was never a separate thing that happened to my plug. Neither was "unknown."

(The POWERON row also moved from 4 to 5, because the plug genuinely lost power once more on September 16 — the blip that came back with the relay off. That one's just the passage of time, not an error.)

Where the second name came from

Not from the device. From me, on March 5th, at 9:09 in the evening.

commit 340a524  "Spring safety tuning: multi-recipient alerts,
                  corrected reset reasons, post-reboot awareness"

- reset_names = {1: "POWERON (power loss)", 3: "SW_RESET (firmware crash)",
-                4: "OWDT_RESET (watchdog timeout)"}
+ reset_names = {
+     1: "POWERON (power-on)", 3: "SW_RESET (software/panic restart)",
+     4: "PANIC (CPU exception)", 5: "INT_WDT (interrupt watchdog)",
+     6: "TASK_WDT (task watchdog)", 9: "BROWNOUT (voltage drop)",
+ }

My original dictionary had three entries and one of them was wrong. Code 4 in the ESP-IDF reset enum is ESP_RST_PANIC, not a watchdog timeout. Code 5 wasn't in my table at all, so the one that arrived on March 4th fell through to the fallback and got written down as "unknown (5)."

I noticed and fixed it. The commit message literally says corrected reset reasons. I did the right thing in March.

And then the log kept both. Four events before the fix carry the wrong name permanently, because the name was baked in at write time. Six months later I read my own log back as data, counted the strings, and published a distribution containing a mistake I had personally identified and repaired half a year earlier.

Why this shape is dangerous

A mislabelled row is trivial on its own. What makes it worth a post is how the error presents when you go looking for causes.

"OWDT_RESET" appears 4 times, from February 26 to March 5, and then never again. On a timeline that reads as a failure mode that started, ran briefly, and got fixed. It sits right at the front of the crash history, ending the day before the March crash loop begins. If I had gone hunting for what changed in early March, that row would have looked like a lead, and it would have been a lead to nothing, invented entirely by an edit to my own source code.

Stale labels don't just add noise. They manufacture structure: clean start dates, clean end dates, modes that appear and vanish. That is the exact shape of evidence you're looking for when you're diagnosing something, which is why it gets picked up instead of discarded.

This is the same failure I keep finding in this system in different costumes. The watcher I told the answer to in its prompt. The failsafe nobody could arm. Every time, the system is confidently reporting something that is really a fact about its own configuration, not about the pump.

What to do about it

This is cheap to prevent and annoying to fix later.

1. Log the raw value next to the label, always. reset_reason=4 (PANIC) costs six characters and makes the line permanently re-interpretable. Mine does this now, which is the only reason I could recover the true counts at all. Had I logged the name alone, those four events would be unrecoverable.
2. Count on the raw value, never the string. If your analysis greps for human-readable text, it is analysing your source history as much as your equipment.
3. Decode at read time where you can. A log line is append-only and permanent; a lookup table in your reporting layer can be corrected retroactively for the whole archive.
4. When you fix a decoder, grep the archive for the old strings. The March commit should have come with two minutes of "what did I already write down wrong?" It didn't, and that omission is what cost me the table.
5. Treat a mode that cleanly starts and stops as suspect until you've checked your own commits. Real failure modes rarely have crisp edges. Configuration changes always do.

The generic field-failure checklist — water, heat, power, vibration — is about the device. This one isn't about the device at all. My plug never once reported anything false. Every integer it sent was correct, including the four that got written down under the wrong name. The defect lived entirely in the six-character gap between what the hardware said and what my software wrote down about what it said.

That gap doesn't show up in an enclosure rating, and no amount of model accuracy touches it. It's just the boring, load-bearing plumbing between a sensor and a conclusion, which in my experience is where most of this actually goes wrong.