My AI Watcher Reads One Hour of Log and Reports Seven. Its Errors Run One Way.

Hackaday reported yesterday that the Linux kernel has stopped requiring contributors to tag commits that an LLM helped write, and that a bot called Sashiko now reviews patches. In an August mailing-list post, Jakub Kicinski said budget had been secured for Meta's models to review patches and edit commit messages, and, in Hackaday's paraphrase, that they are running multiple frontier models on each patch to weed out confabulations.

The detail I keep returning to is older. When Roman Gushchin, a Google kernel engineer, announced Sashiko in March, The Register reported how it had been tested: against 1,000 recent upstream issues selected from Fixes: tags, bugs with known answers. It found 53 percent of them, every one of which human reviewers had missed. The false-positive rate was "harder to measure," somewhere under 20 percent. Before the maintainers leaned on the reviewer, somebody graded it against things that were known to be true.

I have an AI reviewer on my sump pump. It has written 533 notes since June, four a day. Yesterday I graded its recommendations and found the one that was right was the one it said least. Until this morning I had never graded a single one of its observations against the log it was observing. So I took the last seven days and did.

Thirty notes, seven days

The setup, for anyone new: a smart plug meters a 1/3 HP sump pump. A deterministic monitor runs the relay on a duty cycle. A "smart assessor" reads the monitor's log every fifteen minutes and classifies the situation. A guardian polls the plug every two minutes as a backstop. And four times a day, at five past midnight, six, noon and six, a scheduled job runs a language model with shell access, tells it to read the logs, reason about the gestalt, and append a two-to-four sentence note to a file.

Between October 9 and this morning it wrote thirty notes. For each one I took the six hours back to the previous note as the window and computed the ground truth from the three logs: how many pump pulses, the range of end-of-pulse watts and temperature, the guardian's temperature range, every failed read of the plug, every discovery scan, every guardian cutoff, every float-unstick attempt. Then I did something I should have done in June. I opened the transcript of each run and read what the model had actually been shown.

What it reads

The prompt says: "Read the last ~30 lines of each" of the three logs. It did exactly that. Twenty-seven of thirty runs ran tail -30 on the monitor log; the other three ran tail -40. No run opened a file any other way. Five of thirty ran a grep across the monitor log for something specific.

Thirty lines of the monitor log, in the state the pump has been in for ten days, is five pulses. The transcripts confirm it: the first and last monitor timestamps each run saw span 55 minutes or less. The 6:05 pm run on October 15 saw 5:14:49 pm through 6:04:52 pm. Thirty lines of the guardian log is one hour. Thirty lines of the assessor log, which writes one line every fifteen minutes, is seven and a half hours. On top of those it gets the assessor's own one-line summary, which the assessor computes over the last 400 monitor lines, about thirteen hours.

So each run sees one hour of the two logs that record events, seven hours of the log that records verdicts, and a thirteen-hour summary with six numbers in it.

What it says it read

"Reviewed ~7h of logs" opens twelve of the thirty notes. Two say "~10h." One says "~14h." One says "over the full 24h window reviewed." One says "over last 24h." One says "over the past ~12h." Three say "~1hr."

The seven is real. It is the span of the assessor log's thirty lines, applied to all three files. The model was given one hour of events and seven hours of verdicts, and it filed the result as seven hours of events.

Seven failed reads, graded

The cleanest thing to grade is the plug going unreachable, because it leaves a line in a log every time. There were seven such episodes in the seven days: fifteen HTTP 000 errors from the monitor across five of them, and three cases where only the guardian's subnet scan came up empty. Here is each episode, whether the evidence was in the model's inputs for the next run, and what the note said.

Two right, one half right, four missed. Two of the four misses are the interesting kind: the number five was in front of the model, in a line it was told to read, and the note it wrote says "stable" and "in sync." The longest outage of the week, six and a half minutes of a pump controller unable to see its pump, is one of those two.

The other two misses are the boring kind, and the boring kind is the finding. The events happened outside the hour the run was shown. The notes do not say "nothing in the last hour." They say "reachable throughout" and "reachable" about a window the run never read. A statement about six hours was manufactured from one, and three of the thirty notes did it on this topic alone.

What it could see, it read correctly

This is the part that keeps it from being a story about a bad model. When I compare each note's temperature range against the maximum temperature that was actually in that run's inputs, twenty-eight of thirty are within one degree. Three notes match the log to the watt or the tenth of a degree: "468-504W" on the 13th, "42.7-47.7C" on the 14th, "478-511W" this morning at midnight. Every unstick attempt it names, four of them, matches the log to the second and the watt.

But compare the same notes against the six-hour window they claim, and ten of thirty understate the maximum temperature by a degree or more. None overstate it. Nine of thirty give a watt range narrower than the log's. None give a wider one. The week's hottest reading, 48.0 °C on the evening of the 14th, was reported as 47, because 48.0 happened outside the hour.

That is not confabulation. It is an accurate reading of a one-hour sample, labeled as the six-hour window it was asked about, and the labeling error only ever runs in one direction. A sample of an hour will usually miss the extremes of six. So the watcher's week, read as a whole, is calmer than the pump's week, and it is calmer in every column.

The sentence it copies

Seventeen of the thirty notes say the float is "not stuck." Eight say it is "opening normally," "opening on its own," or "self-opens each cycle." Thirteen cite the assessor's counter float_opens=0 as the evidence for this.

Here is the week from the monitor's side. 901 pulses. Every one ended by the controller's 120-second timer. Zero lines saying the pump stopped on its own. In this state the relay cuts power before the float could ever open, so the float cannot be observed doing anything, and the counter that would record it is structurally zero. The watcher read a zero and wrote "opening normally," thirteen times.

It did not invent that. The assessor's own verdict text, the thing printed to the model every run, is a template I wrote on June 2:

Recent runs draw a healthy ~501W (wet pumping) at 42.8C, and the float has opened on its own 0x in the window (so it is NOT stuck).

That sentence has been printed 13,120 times. The conclusion is hard-coded; the counter that contradicts it is interpolated into the middle. And the prompt says it a second time, in its own voice: "healthy ~500W wet cycles at moderate temp with the float opening on its own = NOT stuck." Across all 533 notes since June, 181 say "not stuck" and 93 say the float is opening. The model was handed a conclusion with a zero in it and repeated the conclusion. I would have been surprised by anything else.

Rates, verdicts, dates

The float-unstick routine ran six times a day, every four hours and two minutes, all seven days. Forty-two attempts, zero successes. The October 12 note says it runs "steady ~12/day." The October 14 note, after listing three attempts in nine hours, says "happens ~3x/day currently." The October 13 note says three attempts "each correctly concluded 'not stuck' ... system is correctly distinguishing inflow from stuck float, no fix needed." The three log lines read UNSTICK FAILED: pump still running. The routine's verdict, by its own test, was "still stuck," and the watcher reported the opposite as the system's conclusion.

The October 12 note dates the current episode to September 28. This morning's note says "known since 10-06." The actual re-entry was 1:16 am on October 6, after a normal stretch on the 4th and an eighteen-hour lockout on the 5th. The reason the date wanders is that the watcher has no history, except that it does, and nobody knew.

The diary

The transcripts showed four of the thirty runs reading a file I had never seen: a memory file the model keeps for itself, 25 kilobytes, thirty dated entries since September 15, and a section headed "Standing human TODOs" with eight items. Item 5 is the assessor's parser bug at line 179, written there on September 15, the same day as the reasoning-log note from yesterday's post. Item 7, from September 27, reports that the assessor's classifier crashes with a TypeError when there are no recent runs to take a median over. I have not verified that one. It is not in my fix queue. The file still gives the plug an address it left on September 25 and describes the stack as running "on this Mac," which it does not.

So the watcher's history lives in two files. Nothing on the host reads either. Four of thirty runs read the second, and the "known since" date changes depending on whether the run was one of the four.

What changes

Five things, none of which need the monitor restarted.

1. The assessor's template loses "(so it is NOT stuck)." When the counter is zero in a timed tier, it will print "float not observable in TIER_3, 0 self-stops in window," which is what the number means.
2. The prompt will say what the thirty lines cover: about an hour for the monitor and guardian, about seven for the assessor. And it will say: state the span you read, and do not assert anything about hours you did not read.
3. The grading I did by hand this morning becomes a script that runs weekly beside the liveness audit: for each note, the ranges and events it claims against the log for its own window, and against its own inputs. The difference between those two tables is the whole finding, and it should be a column, not a blog post.
4. I read item 7.
5. The watcher either reads its memory every run or the memory goes. Four of thirty is the worst of both.

For the record: in the same seven days the watcher sent zero alerts, and nothing warranted one. Its one coherent recommendation, to skip the periodic unstick when recent pulses already show 500 watts, is coherent. The problem was never that it cries wolf. The problem is that its reassurance is drawn from a sample a sixth the size of the window it reassures you about.

Three questions for any model over your telemetry

The kernel's reviewer earned trust by being measured against a thousand known answers. A building monitor's known answers are cheaper than that: they are the raw logs the model is summarizing, and a week of them contains a handful of hard events that either appear in the summary or do not. If a vendor puts an LLM between your sensors and your inbox, ask:

• What did it read? In lines and in hours. If the answer is "the recent logs," ask for the tail length.
• What does it say it read? Compare the two. Mine said seven and read one.
• Which way did its errors run last week? Errors that run toward calm are the ones you will not notice until the day they matter.

If nobody can answer the first question, the second is decoration, and the third has never been asked.