Live video is a different inference workload. Here's why.

Every external benchmark and incident figure below comes from a linked public source, reviewed on 8/24/2026.

Physical AI should be easy. Hook up a camera to a VLM, and boom: real-world automation. But for anyone who's tried it, the reality is pretty different. Wiring live video up to AI doesn't work well. Queries are slow. Memory steadily climbs and doesn't recover. As things slow down, inference falls behind the camera, and your reasoning goes out of date.

And it's not obvious why. The GPU idles while the CPU melts. What should be easy isn't, and most teams we've talked to assumed it was their own bridge code.

We wanted to understand what's going on. So we dove in and googled every public measurement, issue thread, RFC, and benchmark we could find on live-video VLM serving, across vLLM, SGLang, TensorRT-LLM, and even the cloud realtime APIs.

These are all excellent tools, built and maintained by sophisticated teams. What we found is not a story about bad software. It's a story about applying AI to a new type of workload, and the challenges are fascinating (if you're into that kind of thing). Here are the five places where live video and general-purpose serving diverge.

The five mismatches: video plumbing that starves the GPU (a 13× throughput loss), prefill that re-reads the whole stream on every question, CPU-bound video decode, memory that drifts until OOM, and cold starts measured in minutes.

1. Video plumbing starves the GPU: a 13× collapse

vLLM's own multimodal benchmark (vllm bench) shows how sharply throughput collapses. Issue #24728 measures throughput falling from ~132 tok/s on a single image to ~10 tok/s on a 30-frame video. That's a 13× loss. Even weirder: the CPU is pegged at 100% while the GPU idles at ~20%. This isn't a GPU inference problem; it's a video plumbing problem. But where, exactly?

The author of a long-video issue (#22695) describes "very high peak GPU memory" and a "long 'dead time' during the pre-fill phase." The issue was later closed as "not planned" (it's marked done in vLLM's multi-modality project tracker, so related work may have landed elsewhere). That's normal open-source triage, not neglect; maintainers have to prioritize, and live video is not the workload these engines are tuned for. A community-proposed streaming-video PR (#39642) has been open for four months and has yet to merge. The RFC for real-time video (#38141, which set the sensible robotics bar of 2–8 fps at <500 ms) was closed without shipping. The pattern isn't anyone failing; it's just that their priority is (correctly) on the datacenter workload.

2. Prefill re-watches the whole tape on every question

Today's AI inference engines answer a question about a live stream the way a security guard would if they re-watched the entire tape from the beginning every time you asked something. "Did anyone come in just now?" Rewind to the start, re-watch everything, answer. Ask again two minutes later: rewind, re-watch, answer. That's how general-purpose engines work: they prefill the entire video before inference begins.

That's obviously inefficient and slow. ElasticMM, a NeurIPS 2025 paper, put numbers on what decoupling buys you: by separating vision encoding from generation and overlapping the two, it cut time-to-first-token by up to 4.2× and lifted throughput 3.2–4.5×. Neat, but it gets there with elastic pools of datacenter GPUs, which is hard to pull off on edge-class hardware.

I know what you're thinking: "just show it less video." After all, some questions don't need much context. "Is the door open right now?" just needs the freshest image. But a lot of questions about streaming video are about change: did the worker just pick up the box, where did the forklift go, is the arm drifting. Change needs the last few seconds, not one image. The right amount of past usually turns out to be small (a few recent frames often beat long histories; more on that below). But the problem isn't really the size of the past. It's paying full price to re-read it on every single question.

A better design does the watching ahead of time, as frames arrive, and keeps a small recent window warm. That way questions (i.e. inference) can kick off right away, instead of reaching for the rewind button every time.

3. Video decode eats CPU and corrupts under pressure

A vLLM RFC (#41843, still open) put numbers on the cost of decoding video on the CPU: moving decode to the GPU delivered 1.88× request throughput, 4.4× lower mean time per output token (TPOT), and ~6× lower CPU usage. And when pipelines get backpressured, the tempting move of dropping encoded packets corrupts the H.264/H.265 decompression. Your model then reasons over smeared garbage, silently. Less than ideal.

The fix NVIDIA endorses on that same thread is a post-decode latest-wins queue: decode everything, keep only the N freshest frames. None of the general-purpose engines implement this. Understandable, since it's a video-pipeline concern, not a text-serving one.

Robotics infrastructure has the same problem. With the Robot Operating System 2 (ROS 2), teams watch cameras arrive at the model at a fraction of their native frame rate because of Python-side copies and serialization. Not bad code, just more copies than anyone needs.

4. Leaks, drifts, and crashes

Physical AI systems have to run for days, not minutes. Stability is still the hard part. Production reports describe memory climbing ~0.1 GB per inference iteration until OOM (in vLLM's Ascend NPU port, traced to cache-reservation logic the port had dropped); a multi-day vision-memory leak spanning multiple releases with, in the reporter's words, "no acceptable solution"; and a production pipeline dead after ~7 hours, with one RTSP source drifting from ~29 ms average latency to ~88 ms while its siblings stayed flat.

Before serving, vLLM runs a "profiler" pass to budget GPU memory: it has to guarantee that no request ever runs out of room mid-answer. The profiler uses worst-case multimodal inputs, and a Qwen3-VL report shows an encoder-cache budget of 16,384 tokens profiled against one maximum-feature-size item. That's a huge video. Your live-video frame window might need only a few hundred tokens, but you pay for the full reservation anyway. A reasonable datacenter default, with surprising consequences at session scale.

An 8-hour unattended demo is, today, something almost nobody can show.

We find that fact weirdly motivating.

5. Cold boots take minutes, not seconds

Published measurements peg cold starts at roughly three to eight minutes. Microsoft's container study measured vLLM engine startup at ~176 seconds regardless of how the container image was pulled. An independent benchmark of vLLM on an A100 GPU recorded 460 seconds with a default configuration, and the startup logs show where the time went: 164s compiling GPU code (torch.compile), 94s on the engine's profiling and warm-up run, and 44s recording CUDA graphs (pre-built execution plans that speed up steady-state serving). Loading the model weights themselves took just 12s. The rest, over five minutes, was setup before the first request could be served.

When it's one server in a fleet, this is an inconvenience. On a robot that power-cycles between shifts, it's a deal-breaker.

Why not cloud?

Cloud works when network latency, connectivity, budget, and data-handling requirements allow it. But some Physical AI deployments need offline operation, local camera processing, or tighter latency control. Those workloads push inference toward the edge.

Why do the engines struggle?

Today's engines fall short on Physical AI workloads. Not because of "bad code" or anything nefarious. These engines are designed for datacenters, serving thousands of concurrent text requests at high batch throughput, and they are very, very good at it.

Multimodal arrived later, as an add-on. On TensorRT-LLM's own issue tracker, a December 2025 user issue titled "The multimodal feature in TensorRT-LLM is just a demo" sits open, alongside months of unanswered Qwen3-VL support requests.

Live video wants the opposite design center: low batch (often just one), bounded state, and worst-case guarantees on one GPU. Getting from one to the other is more than a refactor, as all the issue threads above show.

Accuracy research points the same way. SimpleStream shows a 4-frame sliding window on a stock VLM matching or beating elaborate streaming-memory architectures (67.7% on OVO-Bench), and finds a perception–memory trade-off: adding history can help recall, but it often makes real-time perception worse, not better. StreamingVLM sustains 8 fps real-time commentary on a single H100 with a small recent-window policy. The physics of the problem point to a small, hot, always-fresh window: exactly the shape a single-GPU engine can handle. It's rare that a constraint hands you better results. We'll take it.

What we're doing about it

We're building Photon to be the Physical AI inference engine. Photon 2.0 introduced a new compiler-powered inference technology that beats or matches vLLM and SGLang on single-GPU, low-batch workloads. In matched ChartQA benchmarks on H100, Photon won every throughput test at batch sizes 1 through 8, and beat both engines on cold-start time for every supported model.

Our compiler is novel in another way: it generates "megakernels," so the inference work runs entirely on the GPU. That means:

1. It runs faster, especially at low batch counts, where CPU–GPU chattiness usually dominates.
2. It saves CPU cycles and cuts CPU–GPU bus traffic.
3. It uses less GPU memory, leaving room for more models on one GPU.

We're excited by these results, and we're now scaling the approach across more models and more hardware.

We've started work on Photon Live Video: session-based live-video serving for small VLMs/VLAs on one GPU. Our targets:

These are design targets, not current performance claims.

We're taking three design partners. If you have a live camera workload, a GPU next to the machine, and a deployment deadline, we want your workload as our spec: early access, a direct line to our Photon engineers, an SLO matrix tuned to your model and hardware, and a co-signed soak report on your feed.

Not ready to book? Follow @moondreamai on X to watch Photon Live Video come together.