Why do we need a VLM Inferencing Harness?

A camera can record a delivery, a vehicle at a gate, or someone trying to force a door. But someone still has to notice, work out what matters, and do something about it. I want to make that possible without asking a person to watch a feed all day.

The models are giving us more to work with:

• Qwen3-VL combines video understanding, reading text in images, and locating objects. Its model family spans sizes intended for edge and cloud deployment.
• Google’s Live API supports ongoing visual input and tool use. It shows how a model can participate in a live session, rather than only answer questions about an uploaded file.
• NVIDIA’s Cosmos edge recipe connects camera streams, frame sampling, local inference, and structured results on Jetson hardware.

My bet is that these capabilities will keep improving. I want Nurby to benefit from those improvements without having to rebuild the camera setup, the rules, and the integrations every time a better model arrives.

That’s how I describe the idea to myself: a “Claude for CCTV.” You tell it what you want to watch for, give it access to the relevant tools, and let it follow what’s happening as the day unfolds.

For example: “When a vehicle arrives, read its plate, check whether it’s scheduled, and open the gate if it’s expected. Otherwise, call the supervisor.”

A model can help interpret the image. It still needs software to retrieve the schedule, keep track of the arrival, decide which action is allowed, and confirm whether the gate actually opened. That surrounding software is what I mean by a VLM inferencing harness.

Nurby is my chance to build it. The Claude comparison describes the experience I want: an assistant with context and tools, specialized for watching things happen.

The design starts with local video processing and plain-language rules. I want to connect those to a system that can remember events and use tools reliably. Here’s how the pieces fit together.

1. 01

   Camera feeds

   Connect existing cameras. Keep footage on the local network.

2. 02

   Local inference

   Choose frames and short clips for the model to examine.

3. 03

   Event context

   Keep track of what happened, where, and when.

4. 04

   Rules and tools

   Check the schedule, apply the rule, and send the right action.

The model and runtime need to fit the hardware, cameras, and workload. I want to be able to use a better model when it becomes available, while keeping the rules and connected tools in place.

These are the workflows I’m building toward. Each needs camera observations, outside context, and a clear rule for what happens next.

Follow a delivery, a scheduled vehicle, or an after-hours event through the workflow. Choose an example or a step to see how an observation becomes an action.

A convincing example is a start. I also need to know how often the system misses an arrival, sends an unnecessary alert, or takes too long to respond. Night footage, a partially hidden plate, and several people at a door are part of the job.

Those are the checks I want to build around: missed events, false alerts per camera-hour, time from event to action, and whether the connected tool completed the request. When comparing models, I care about those results on the same footage and hardware.

This is where my experience with agent platforms and integrations matters. I’m working on the product decisions around the model: what people can ask it to do, what context it needs, and how they can check what it did.