Anna Kriuchkova

Case files / 02

The camera detects. A second model decides.

Our phones were being told about ducks that were cats and birds that were the front door. Rebuilding the feed turned up something worse: one part of the system had been failing for days, and its failure looked exactly like a quiet week.

Problem
Notifications built on a detector's raw guesses, and a wildlife log that stayed empty while geese stood in plain view.
Evidence
One real event, run through the pipeline by hand, with the error message read in full.
Fix
Every animal passes a second model before it reaches anyone. An error can no longer pass for "nothing there".
Result
A feed that carries only what was verified, and a log that fills when there is something to log.

First, a sample that lied

The session started with a smaller complaint. A known car had left the driveway in the morning and the house had not announced it. The AI coding agent investigating looked at recent camera events and reported that plate reading was broken: no plate had ever been read.

I did not believe it, because I had seen it work. The agent had sampled only the most recent events, and that sample was full of the previous evening's traffic. Asked again for the morning window, the same system showed the plate had been read and matched correctly, to the second. The reading worked. Nothing downstream was using it: a search of every automation for that field returned nothing at all.

The fix was a small one, an announcement driven by the verified reading. The lesson was larger, and it set the tone for the rest of the day: a convenient sample is not the data.

A cat, announced as a duck

The next complaint was funnier. An alert had confidently described a cat as a duck. Meanwhile the wildlife log, which should have been recording the geese and an egret that visit the yard, was completely empty.

Instead of reading configuration, I had the wildlife pipeline run by hand on one real bird event, with every response shown. The vision model had not answered "no animal". It had answered with an error.

Why every single request failed

  • Request size4,163 tokens
  • Limit configured4,096 tokens
Each image was 67 tokens too large for the limit the script had set. The model refused every request, and the script could not tell a refusal from an empty yard.

The script treated an empty response the same way it treated "no animal": it logged nothing and moved on. So nothing had ever been logged, and nothing had ever complained.

A silent "none" had been hiding a total outage.

Raising the limit fixed the immediate fault, confirmed on the same event: the answer came back "Canada goose". The previous night's sightings were then filled in from the recordings. The more important change was to the script, so that an error is reported as an error and can never again be counted as a negative result.

One cause behind three complaints

The cat announced as a duck, a run of "bird" alerts at the front door, and the trouble with the geese all traced to the same design fault. The automations trusted the detector's first guess. Object detectors are fast and useful, and they are wrong often enough that their labels should never go straight to a person.

My requirement was simple: I did not want the detector's hallucinations in my feed. The design that followed has one rule. The detector detects, and a second model decides. Every animal event now passes through a verification step, and anything the second model will not confirm is dropped before it reaches the log or a phone.

It was checked the only way that counts, on real events: a car at the front door that the detector had called a bird was dropped; the geese were logged and announced correctly.

Then the phones had opinions

The last request sounded trivial: tapping a notification should open the picture of the event. It took three failed attempts. A local address did not work away from home, and one phone refused it for not being encrypted. A relative link was intercepted by the app and opened the dashboard instead. A link to the video clip opened and then would not play.

What worked on both an iPhone and an Android phone was the plainest option: a secure, complete link to a still image. The same thing happened with the picture history on the dashboard. After the platform resisted a proper gallery in several ways, my suggestion was to stop building a gallery and stack a few large, fresh images with captions. That shipped in minutes.

What I took from it

  • Verify on the real event, not on a convenient sample.
  • An error must never be allowed to look like a negative result. Keep "failed" and "found nothing" apart.
  • Decide where judgment lives. The fast component proposes; a slower one confirms.
  • When a platform fights the same requirement three times, simplify the requirement.

"Faith" is a fine invention
When Gentlemen can see -
But Microscopes are prudent
In an Emergency.

From my shelfEmily Dickinson, Collected Poems

Previous: The rock that was a birdNext: Twenty times faster

Contact

Have data that needs a second look?