SAM 3: Segment Anything with Concepts

1. Encode the image and the concept

The shared Perception Encoder (PE) vision transformer has ~450M parameters and 32 layers. It processes 1008 × 1008 images using mostly local attention in nine 336-pixel windows; four layers use global attention. Image features are computed once per frame and reused. The ~300M causal text transformer encodes up to 32 tokens, aligned with visual features through image-text pretraining.

An exemplar combines a box’s position, its positive/negative label and ROI-pooled visual features. A small transformer encodes it. Text and exemplar tokens jointly form the prompt. A six-layer fusion encoder lets image features attend to that prompt, highlighting evidence relevant to the concept.

2. Decode objects and check concept presence

A six-layer DETR-style decoder uses 200 learned object queries as candidate instance slots. They attend to each other, the prompt and the conditioned image, refining boxes at each layer. A MaskFormer-style head predicts instance masks, with SimpleFPN supplying feature scales. A semantic head also predicts all pixels matching the concept.

A separate global presence token decides whether the concept exists anywhere in the image. Each object query estimates its match conditional on presence; the final confidence multiplies the two:

This separates global recognition from local mask placement. A cup-like patch should not get high confidence when the requested concept is absent. Negative image-phrase pairs train the presence head; per-query classification is supervised on positive pairs. A two-expert ambiguity head adjusts classification logits for competing phrase interpretations while sharing masks and boxes.

3. Track identities through video

On each frame, the detector finds concept matches while a SAM 2-style tracker propagates existing objects from per-object memory. IoU matching associates detections with propagated masks. New detections start new masklets, meaning sequences of masks for one object. Temporal confirmation removes spurious or duplicate tracks, and reliable detections periodically refresh the tracker to reduce drift.

The tracker predicts three candidate masks and chooses the most confident, with an occlusion head estimating visibility. It stores confidently present frames in memory. The default confirmation delay is 15 frames, and periodic re-prompting occurs every 16 frames when confidence and overlap pass thresholds. Detection treats cups as equivalent concept matches; tracking must preserve each cup’s identity.

The image training unit is (image, noun phrase, complete set of matching masks). A positive “mug” pair needs every mug’s mask. A verified “wine glass” negative has an empty mask set. An omitted mug is a missing annotation, not evidence of absence, which makes exhaustivity central to training.

Annotations include instance masks, group masks for inseparable crowds, phrase-level presence or absence, and completeness judgments. Video adds consistent object IDs and masks across frames. Human correction can add, remove or repair masks and tracks. Difficult video collection also permits partially exhaustive clips when some objects cannot be annotated.

Tables 24–25. Counts are phrase-associated annotations; one object may match multiple phrases. Negative percentages refer to media-phrase pairs, not images.

HQ uses human or AI verification, rather than exclusively hand-drawn masks. SYN means automated labels on real MetaCLIP images, not generated images. EXT combines external annotations, including box-only labels converted to masks by SAM 2, and is partially exhaustive. VIDEO is human verified. SA-1B and other video segmentation datasets additionally support visual-prompt training.

The 22.4M-node vocabulary is a Wikidata ontology for collecting data, organized into 17 top-level categories and 72 subcategories. It is distinct from the 4M noun phrases in HQ and from the text tokenizer. SAM 3 accepts a phrase at inference; it does not choose from 22.4M fixed output classes.

The authors select high-level Wikidata nodes and include their descendants. Sentence-BERT retrieves candidate nodes for a phrase, and Llama 3.2 chooses the closest match. This guides image retrieval, coverage balancing, synonyms and hierarchical consistency: an annotated canoe must also count as a boat when both labels are used.

The ontology also proposes hard negatives through related nodes: “Siamese cat” might suggest “tabby cat” or “dog”. Llama 4 proposes visually similar alternatives too. Candidates are useful when SAM 3 mistakenly segments the known positive object. Verification still matters because an alternative, such as a dog, could really appear elsewhere in the image.

Mask verification checks phrase relevance and boundary coverage for each candidate. Exhaustivity verification checks whether matching instances remain unmasked. Humans supply discrete quality/completeness ratings and correction examples. More than 200M image-text pairs pretrain a foundational verifier, followed by roughly 10M high-quality human examples to specialize the verifiers.

Those are separate data-production models, not LLMs inside SAM 3. The paper does not specify every Llama helper’s exact parameter-size checkpoint. Table 20 publishes category counts, not every node or training phrase; open vocabulary also does not guarantee reliable recognition of every possible concept.

1. Pretrain the image and text encoders

PE-style contrastive learning on 5.4B image-text pairs aligns matching visual and language representations. This supplies semantic coverage before segmentation. SAM 3 omits PE’s distillation and video fine-tuning in this stage.

2. Pretrain the detector and both encoders

Train jointly on HQ, SYN, external segmentation data and video frames treated as images. PCS learns boxes and masks; visual-prompt data also trains PVS using four extra decoder queries. Phrases are replaced with visual queries with probability 0.2, or augmented with exemplar boxes with probability 0.2. PVS allows four interaction steps.

The run uses AdamW, ~95K iterations and batch size 896, with a reciprocal-square-root schedule. Initial learning rates are 5 × 10⁻⁴ for vision, 1 × 10⁻⁴ for text and 1 × 10⁻³ elsewhere. Box L1/gIoU loss weights are 5/2, Align classification is 100, and mask focal/Dice weights are 200/10. These teach object assignment, box extent and pixel coverage.

3. Fine-tune precision and interactivity

Retain the highest-quality, exhaustivity-verified data and drop SYN. Add the presence token with binary cross-entropy weight 20. Train for ~5K iterations, batch size 512, reducing learning rates to 0.025 times stage 2.

PVS increases to seven interactions. PCS gets five simulated correction rounds: missed objects supply positive boxes; high-confidence false detections supply negative boxes. Prompts accumulate, teaching the model to respond to its own errors.

4. Train video tracking on a frozen backbone

Freeze the visual backbone to preserve grounding while training the tracker for ~190K iterations, batch size 512, with a cosine schedule and peak learning rate 5 × 10⁻⁴. Loss weights are 20:1:1:1 for mask focal, Dice, IoU-prediction MAE and occlusion cross-entropy. Only the best candidate mask is supervised; frames without masks still supervise visibility.

A further 60K iterations use longer 16- and 32-frame contexts with learning rate reduced tenfold. Augmentation includes crops, resizing, random padding, suitable horizontal flips and selected mosaics up to 3 × 3.