Moondream 3.1 9B A2B
Moondream 3.1 is an open-weight vision-language model built for engineers who need pixel-grounded answers, exact box, point, mask, or field they asked for. It has a sparse mixture-of-experts architecture with 9.0B total / 2.0B active per token, reads image + text into a 32K context, ships under Moondream License, runs on Transformers and Photon, and exposes five native skills: query, caption, detect, point, and segment. It also supports “grounded” reasoning (reasoning with special tokens that refer to 2D points in the image).
The instrument readout
All revisions are immutable. Pin revision=md-3.1-9b-a2b-r20260628 in production; floating main may move.
Five typed contracts
Each skill has a fixed input and output contract, so you are not reverse-engineering a prompt. Coordinates are always normalized [0,1] from the top-left corner; multiply by the image width and height to get pixels back. Defaults below are greedy unless a row says otherwise.
Open-ended question answering grounded in the image. The general-purpose entry point.
query(
image: Image,
question: str,
stream: bool = False,
settings: {temperature, max_tokens}
) -> {answer}{
"answer": str, // free-form
"request_id": str
}res = model.query(img, "How many vehicles are on the road?") print(res["answer"]) # -> "Four."
Generate a description of the whole image at three densities.
caption(
image: Image,
length: "short"|"normal"|"long" = "normal"
) -> {caption}{
"caption": str
}model.caption(img, length="short")["caption"] # -> "An aerial view of a rural road with parked cars."
Open-vocabulary object detection. Returns axis-aligned boxes for any phrase.
detect(
image: Image,
object: str, // natural-language target
threshold: float = 0.30
) -> {objects}{
"objects": [
{x_min, y_min, x_max, y_max,
label: str, confidence: float}
]
}for o in model.detect(img, "the red car")["objects"]:
print(o["confidence"], o["x_min"], o["y_min"])Locate the centroid of each matching instance. Cheaper than boxes and ideal for counting.
point(
image: Image,
object: str,
max_points: int = 64
) -> {points}{
"points": [ {x: float, y: float} ]
}pts = model.point(img, "each car")["points"] print(len(pts), "instances")
Instance segmentation: one binary mask per matching object.
segment(
image: Image,
object: str
) -> {masks}{
"masks": [
{rle: str, area: int,
x_min, y_min, x_max, y_max, label}
]
}m = model.segment(img, "the road")["masks"][0] mask = rle_decode(m["rle"]) # HxW uint8
Grouped by what you'll use it for
We grouped the benchmarks by what people actually use Moondream for, not by academic category. Each group spells out the resolution, decoding, and sample count we ran, plus a caveat that says where the number flatters or undersells real behavior. The Moondream 3.1 column is the model in question; the baselines (Qwen3.5 27B, Qwen3.5 9B, SAM 3, LocateAnything, Gemma 4 12B) ran under the same settings unless a note says otherwise.
↑ higher is better unless marked ↓ · detection metrics are F1@0.5 (mIoU where noted) · a blank cell (n/a) means the model cannot do that task · SAM 3 and LocateAnything are detection / segmentation specialists, not general VLMs.
| Benchmark | Moondream 3.1 | Qwen3.5 27B | Qwen3.5 9B | SAM 3 | LocateAnything | Gemma 4 12B |
|---|---|---|---|---|---|---|
| COCO (val) | 81.46 | 76.75 | 75.58 | 77.06 | 70.1 | 65.93 |
| ODinW-13 (mini-val) | 93.94 | 92.37 | 91.94 | 82.16 | 84.33 | 88.1 |
| LVIS (val) | 67.4 | 73.8 | 71.11 | 70.01 | 62.3 | 57.93 |
| Benchmark | Moondream 3.1 | Qwen3.5 27B | Qwen3.5 9B | SAM 3 | LocateAnything | Gemma 4 12B |
|---|---|---|---|---|---|---|
| Dense200 | 74.61 | 61.59 | 48.23 | 63 | 74 | 15.96 |
| SKU-110k (test) | 52.77 | 26.39 | 44.27 | 33.92 | 23.47 | 4.76 |
| CrowdHuman | 74.5 | 77.78 | 64.4 | 82.91 | 71.28 | 65.32 |
| Benchmark | Moondream 3.1 | Qwen3.5 27B | Qwen3.5 9B | SAM 3 | LocateAnything | Gemma 4 12B |
|---|---|---|---|---|---|---|
| RefCOCO-M (val) | 87.1 | 88.71 | 85.73 | 83.7 | 46.21 | 74.04 |
| HumanRef (val) | 70.39 | 88.47 | 84.9 | 27.94 | 82.56 | 73.62 |
| Benchmark | Moondream 3.1 | Qwen3.5 27B | Qwen3.5 9B | SAM 3 | LocateAnything | Gemma 4 12B |
|---|---|---|---|---|---|---|
| DOTA-v2 | 59.63 | 55.46 | 50.36 | 26.09 | 29.94 | 38.36 |
| Benchmark | Moondream 3.1 | Qwen3.5 27B | Qwen3.5 9B | SAM 3 | LocateAnything | Gemma 4 12B |
|---|---|---|---|---|---|---|
| CountBench | 90.35 | 96.1 | 94.66 | 68.58 | 83.78 | 93.8 |
| TallyQA (via query) | 74.86 | 76.97 | 72.22 | n/a | 65.74 | 76.41 |
| pixmo-count (val) | 88.68 | 88.39 | 88.66 | 76.77 | 87.55 | 83.09 |
| Benchmark | Moondream 3.1 | Qwen3.5 27B | Qwen3.5 9B | SAM 3 | LocateAnything | Gemma 4 12B |
|---|---|---|---|---|---|---|
| IFBench (CoT) | 38.67 | n/a | 19.67 | n/a | n/a | 30.67 |
| IFEval | 77.63 | 82.07 | 54.15 | n/a | n/a | 69.13 |
| Benchmark | Moondream 3.1 | Qwen3.5 27B | Qwen3.5 9B | SAM 3 | LocateAnything | Gemma 4 12B |
|---|---|---|---|---|---|---|
| GSM8K | 63 | n/a | 89.46 | n/a | n/a | 89.01 |
| Benchmark | Moondream 3.1 | Qwen3.5 27B | Qwen3.5 9B | SAM 3 | LocateAnything | Gemma 4 12B |
|---|---|---|---|---|---|---|
| ChartQA | 86.01 | n/a | 70.72 | n/a | n/a | 62.24 |
| DocVQA | 88.6 | n/a | 92.94 | n/a | n/a | 81.94 |
| Benchmark | Moondream 3.1 | Qwen3.5 27B | Qwen3.5 9B | SAM 3 | LocateAnything | Gemma 4 12B |
|---|---|---|---|---|---|---|
| OlmOCR | 32.37 | n/a | 33.1 | n/a | n/a | 13.51 |
What went in, and what didn't
We share enough to reason about coverage, bias, and freshness, but stop short of a recipe you could use to rebuild the model or pull training data back out. Exact mixture weights, licensed corpora, and internal filtering thresholds stay private.
Publicly available web images with alt-text and captions, licensed image datasets, open academic detection/segmentation/VQA corpora (COCO, LVIS, Objects365, OpenImages, RefCOCO family, DocVQA, ChartQA and similar), and Moondream-generated synthetic grounding data. No customer or API-inference data is used for training.
Image + text data through May 2026. The model has no live knowledge of events, prices, or UI layouts that changed after that date and should not be relied on for current-affairs grounding.
A substantial share of grounding supervision (boxes, points, masks, structured fields) is model-generated and then verified, both programmatically and by sampled human review. Synthetic data is the primary driver of the dense-detection and structured-output gains; it can also propagate systematic labeling conventions, which the limitations section names.
Human annotation is concentrated in evaluation sets, preference data for post-training, safety red-teaming, and quality audits of synthetic labels, not bulk box-drawing. Annotators worked under documented guidelines with fair-pay sourcing.
Supervised fine-tuning on the five skill contracts, followed by reinforcement learning on verifiable tasks with a focus on detection F1 / IoU rewards, counting exactness, JSON-schema validity, and instruction-following adherence. RL targets task correctness and format compliance, not engagement or persuasion.
Training images filtered for CSAM (hash-matching + classifiers), explicit and graphic-violence content, and detectable personal data (faces in sensitive contexts, ID documents, license plates where isolable). Filtering is imperfect; residual unsafe or identifying content may remain.
Near-duplicate images are removed with perceptual hashing and embedding-cluster dedup, and train/eval decontamination strips anything overlapping the public benchmarks above. Some leakage on web-scraped benchmark images is unavoidable, so treat single-point margins with caution.
No biometric face-identity datasets, no medical imaging diagnostic corpora, no datasets built to recognize named private individuals, and no scraped behind-login or paywalled content. The model is deliberately not trained to identify specific people.
Where it breaks, concretely
Worth reading before you ship. These are the failure modes we hit most often in evaluation and day-to-day use, written plainly so you can build guardrails around them.
- Small text below ~10px at the tile resolution is frequently misread or skipped; up-res and crop the region first.
- Tiny objects under ~12px on the long edge are missed by detect(); use point() on a cropped region.
- Crowded scenes (> 50 instances) lose recall and merge adjacent identical objects; counts above ~50 are unreliable.
- Thin structures (cables, wires, railings) under-segment.
- Ambiguous pointing. For mass nouns ("water", "crowd") or vague phrases, point() returns whichever salient instance it likes.
- OCR hallucination. On blurry, stylized, or partly occluded text, the model can emit a plausible but wrong string with no low-confidence signal to warn you.
- Caption drift. Long captions invent fine detail; never parse them for counts or coordinates.
- Reasoning traces are not guaranteed faithful to the answer.
- Medical interpretation (radiology, pathology, dermatology): not trained or validated for diagnosis.
- Legal / financial determinations from documents without human review.
- Safety-critical autonomy (driving, robotics collision avoidance, weapons) as a sole perception source.
- Surveillance and biometric identification of individuals.
- Identity / person recognition is unsupported and must not be attempted.
- Demographic inference (age, gender, race, emotion) is unreliable and prohibited for consequential decisions.
- Geographic / cultural skew. Western, English-language, urban scenes are over-represented; signage, currency, and objects from under-represented regions come out weaker.
- Language. OCR and prompts are strongest in English and Latin scripts.
- Adversarial images. Crafted perturbations and typographic attacks can flip detections and answers.
- Prompt injection. In UI-agent workflows, text rendered inside a screenshot ("ignore previous instructions…") may be followed. Never let model output trigger actions unsandboxed.
- Spoofed structure. Schema-constrained output is always valid JSON, never guaranteed-correct values.
- Video is not supported natively; sample frames and run them as images (no temporal reasoning).
- Audio , 3D, depth, and point clouds are out of scope.
- Image generation / editing. The model reads images, it does not produce them.
- Multi-image reasoning across many frames is experimental.
Safety is layered, not baked-in
The open weights carry only light safety post-training (refusal of overtly harmful generation requests, no person-identification behavior). They are not a content-moderation system. Where the safety boundary sits depends on how you deploy.
Light safety training only. You must add external moderation on inputs and outputs, rate-limiting, and abuse monitoring. No telemetry leaves your host, which also means there is no safety net but the one you build.
The hosted API and Moondream Cloud add input/output safety classifiers, CSAM hash-matching, abuse rate-limits, and logging-based monitoring on top of the model. Recommended path for most production use.
Safety concerns & harmful outputs: safety@moondream.ai. Security vulnerabilities: security@moondream.ai (see §08).
CSAM & minor safety · non-consensual intimate imagery · violent/graphic content description · person re-identification & surveillance · demographic profiling · prompt-injection via in-image text · OCR-assisted extraction of PII (IDs, cards) · jailbreak via image+text combinations.
Internal red-team + external reviewers ran ~6k adversarial image/prompt pairs across the categories at left. Refusals, harmful completions, and identification attempts were scored; failures fed back into safety SFT. CSAM testing used hash-matched proxies only, under legal supervision.
Prompt injection from rendered text is only partially mitigated. The model can still describe violence and extract visible PII when asked plainly. Demographic guesses, though discouraged, are still producible. None of this is fully solved at the weights layer.
Following the openness norms of Gemma / PaliGemma and the structured safety disclosure of OpenAI / Anthropic system cards: we publish risk areas, method, and residual risks, and withhold exploit-enabling specifics. This appendix is a summary, not a guarantee of safety.
Running it well
Defaults that reproduce the numbers above, plus the knobs that matter in production. Benchmark figures and settings are in the evaluation matrix (§03).
| Skill | temp | max_tok | decode |
|---|---|---|---|
| query | 0.0 | 768 | greedy |
| caption | 0.3 | 256 | sample |
| detect | 0.0 | 200 box | greedy |
| point | 0.0 | 64 pt | greedy |
| segment | 0.0 | n/a | greedy |
Default tiling adapts 384–1024px, up to 12 crops. Use 768px for VQA/caption, 1024px for detect/point/segment, 1536px for documents. For small targets, crop-then-call beats raising global resolution. Aspect ratio is preserved; extreme panoramas should be split.
Throughput peaks around batch 8–16 on a single 24GB GPU at 768px. When you run several skills against one image, cache the vision-encoder output: it is the dominant cost and is reused across query, detect, and point on the same frame.
Moondream 3.1 ships as a single fp8 build: the mixture-of-experts weights are fp8, with the attention and routing layers kept in fp16 for stability. It fits in about 10 GB of VRAM with headroom for activations and KV cache. There are no separate int8, int4, or GGUF weight variants to pick between.
# pip install moondream
import moondream as md
from PIL import Image
# local=True runs Moondream on Photon — same API as Cloud
model = md.vl(api_key="YOUR_API_KEY", local=True)
image = Image.open("scene.jpg")
out = model.detect(image, "the red car")
print(out["objects"]) # [{x_min, y_min, x_max, y_max, ...}]from transformers import AutoModelForCausalLM
import torch
model = AutoModelForCausalLM.from_pretrained(
"moondream/moondream-3.1-9b-a2b",
revision="md-3.1-9b-a2b-r20260628", # pin it
trust_remote_code=True,
torch_dtype=torch.bfloat16,
device_map="cuda",
)
out = model.detect(image, "the red car")
print(out["objects"]) # [{x_min, y_min, ...}]License, lineage & contact
@misc{moondream2026_31,
title = {Moondream 3.1: An open vision-
language model for grounded
visual reasoning},
author = {Moondream and M87 Labs},
year = {2026},
note = {Model revision
md-3.1-9b-a2b-r20260628},
url = {https://moondream.ai}
}- Prompt injection from in-image text partially mitigated.
- CrowdHuman / person-dense recall trails SAM 3.
- GSM8K-style text math below peer LMs.
- 3.1 · better grounding, dense-detection gains.
- 3.0 Preview · MoE decoder, 9B total, 2B Dense, added segment() skill
- 2.x · dense model lineage, added point() skill
- Revisions are immutable & remain downloadable indefinitely.
- Hosted-API minor versions supported ≥ 12 months after a successor ships.
- Breaking contract changes only on a major version, with migration notes.
Security vulnerabilities: disclose privately to security@moondream.ai (PGP on the site). Please allow a 90-day coordinated-disclosure window before public release.