Modern AI inference infrastructure is built for chat. Engines like vLLM and SGLang are really good at queuing prompts, in large batches, and maximize tokens per second across GPU fleets. That's exactly what agentic workloads need.
Physical AI is different. Robots, cameras, industrial systems, and computer-use agents continuously ingest images, audio, and video. They run with low concurrency, and will sacrifice throughput for quicker responses. They worry about strict latency deadlines, limited memory and several models sharing a GPU. Serving live perception is a different problem. It needs a different inference engine.
Today, we’re releasing Photon 2.0, our next step toward building the inference stack for physical AI. Photon began as the fastest inference engine for Moondream. Photon 2.0 runs Moondream, Qwen 3.5, and Gemma 4 on NVIDIA H100. In matched benchmarks, Photon outperformed vLLM and SGLang in every throughput test and brought every model online faster.
We built Photon 2.0 for a different purpose, and we took a very different approach. Let's explain why.
Hand-tuning doesn't scale
Modern inference stacks are built from large libraries of hand-written and hand-tuned GPU kernels. The results are remarkable. With contributions from hundreds of developers, vLLM and SGLang have pushed open inference forward at an extraordinary pace. But for Physical AI, the combinations are enormous:
Every model × every chip × every deployment objective
Physical AI solutions often use numerous models, and run on a wide set of chips (edge, on-prem, cloud). The objectives often are different too. One deployment needs maximum throughput,another needs the lowest possible p99 latency, or another needs to fit three specific models into limited memory, or with a strict power budget. Writing efficient GPU inference code is hard, and hand-optimizing all of these combinations is practically impossible. Thankfully there's a better approach: a compiler.
We took the knowledge we had from building Photon for Moondream and turned it into a generalized inference compiler. We added support for Qwen and Gemma to prove the compiler generalized, and the results were amazing. This approach is exciting because it opens the door to so many possibilities. This launch is just the first step.
Megakernels
A key technique behind Photon 2.0's performance is called a 'Megakernel'. We all know inference happens on a GPU, but in practice, it's typically done by having the CPU manage a large, tightly coordinated set of GPU pgorams that execute the inference path. There's a lot of 'chattiness' between the CPU and GPU, with inefficient memory bandwith between the two.
Our inference compiler produces a single 'megakernel': one large set of instructions that runs the entire inference on the GPU alone. This reduces launch and synchronization overhead. More importantly, it gives the compiler visibility across operator boundaries, where many of the best optimization opportunities live.
The compiler is winning
Photon 2.0 launches with support for:
- Moondream 2 and 3
- Qwen3.5/Qwen3.6 0.8B, 2B, 4B, and 9B
- Gemma 4 E2B and E4B
The initial hardware supported is NVIDIA H100. We tested Photon, vLLM, and SGLang using the ChartQA benchmark. Photon came out ahead on every batch count (1, 2, 4 and 8 tested).
These numbers measure the full serving system, not isolated kernels, with each engine given the same models and the same request streams. The result we care most about: one compiler produced these speedups across Moondream, Qwen, and Gemma, without per-model hand-tuning.
Models ready sooner
Another area where Physical AI workloads differ: cold-starts. This matters during autoscaling, process recovery, software updates, development, and power cycling. Systems that run outside of large, permanently warm GPU clusters. In our matched cold-start tests, Photon beat vLLM and SGLang on all models supported (note: SGLang doesn't support Moondream).
Available today
Photon 2.0 is ready for production workloads today. It's available through the Moondream package:
pip install moondreamFor example:
import moondream as md
from PIL import Image
model = md.photon("Qwen/Qwen3.5-4B")
# Ask a question
image = Image.open("photo.jpg")
answer = model.query(image, "What's in this image?")["answer"]
print("Answer:", answer)Check out the docs. The Photon inference engine is Apache 2.0, and the megakernels are free to run. Our compiler itself is proprietary.
Roadmap
Photon 2.0 currently supports a narrow set of models, chips, and deployment objectives. We are already expanding that matrix, with new updates every week. Follow @moondreamai on X for the latest, or reach out with specific requests. We are especially looking for design partners building:
- Live video understanding
- Robotics
- Industrial perception
- Computer-use agents
Inference is becoming a compiler problem
Hand-written kernels aren't going away. Assembly never disappeared either. But assembly stopped being the way most software gets built. Inference is entering its compiler era.