My Local AI Agents Are Starting to Grade Their Own Work

In my architecture post I laid out the big picture of how my smart home thinks for itself. This post is a deep dive into one piece of that picture — the part I’m honestly most excited about right now: a growing fleet of specialized local AI agents that quietly do real work for me, and that are starting to learn how good they are at it.

The problem with one big assistant

For a while I had one general-purpose AI handling everything I threw at it — writing config, reviewing firmware, drafting content. It worked, but “works for everything” and “works well” aren’t the same thing. A model that’s decent at writing marketing copy isn’t necessarily the one I want sanity-checking a Home Assistant config before it goes live, and it’s definitely not the one I want anywhere near files I consider private.

So instead of one generalist, I’ve been building out local specialists — 27 of them by now, covering everything from CAD and firmware to security and content. More get added as I find jobs worth splitting off.

Teaching the local agents to grade themselves

The part that actually changed how I work with them is the review loop. Every time one of these local agents finishes a task, I review its output the way I’d review a pull request from a colleague — did it get it right, did it miss something, did it make something up it shouldn’t have. That review turns into a confidence score, tracked per agent and per category of task.

Below a certain confidence threshold, I double-check everything that agent produces before it touches anything real. Once an agent clears the threshold for a category — after enough reviewed tasks — it gets to run that category on its own, without me re-checking every single output. It’s not a one-time “pass/fail” test, it’s an ongoing track record that earns (or loses) autonomy over time.

Proactivity is something I’m actively pushing further, too — right now the local agents mostly act on what I ask, but a few of them already send me unprompted suggestions when they spot something worth flagging. That’s a direction I want to lean into more.

The part that never touches the cloud

Here’s a principle I keep coming back to: some tasks involve data that should never leave my local network — not even anonymized, not even “just this once.” For those, cloud AI is off the table entirely, no matter how good it is. For that kind of work, I run fully local language models instead, on local hardware — no API calls out, no account, no running cost.

A real example: a local agent catching a dirty camera lens on its own

One of my local agents built something I didn’t explicitly ask for in this much detail: a watchdog that checks every outdoor camera on my property for problems, completely on its own. The camera list isn’t hardcoded — it pulls live from my Frigate NVR, which I set up on the same local mini-PC, so any new camera I add gets covered automatically, no extra setup.

It runs a few times a day on my local mini-PC and compares each camera’s current image against a stored reference frame. That catches a camera that’s been knocked out of position or rotated. On top of that it checks sharpness and exposure — which is exactly what caught a real problem recently: one outdoor camera had a dirty lens, flagged automatically before anyone noticed it by eye.

It only alerts me after the same issue shows up twice in a row, and at most once every 24 hours — no alert spam — and it sends an all-clear the moment the problem is fixed. None of this needs a cloud model; it’s pure local image comparison, running continuously without me ever asking it to check.

That’s the part I actually care about most: it’s not just that the local agents are capable — it’s that they’re starting to catch real problems and report them before I’d have noticed myself.

Where this is going

The fleet of local agents will keep growing — new specialists for new recurring jobs, more categories graduating to full autonomy as they earn it, and more proactive suggestions instead of me always having to ask first. I’ll keep writing about what actually works here, including the parts that don’t.