Taming SAM3 under Distribution Shift: $\mathrm{C{\scriptsize ONCEPT}B{\scriptsize ANK}}$ for Support-Assisted Open-Vocabulary Segmentation

🔍 Drift Robustness

We demonstrate ConceptBank's ability to handle real-world distribution shifts where the original SAM3 fails.

1. Concept Drift: Resolving Semantic Ambiguity

Scenario: In the COCO Object dataset, the class "mouse" strictly refers to a computer mouse. Observation:

• Original SAM3: Suffers from Concept Drift. It relies on pre-trained open-world knowledge, incorrectly segmenting the animal mouse because it cannot distinguish the dataset-specific definition.
• ConceptBank (Ours): Correctly aligns with the target domain statistics, segmenting only the computer mouse and ignoring the animal mouse.

Input Image	ConceptBank (Ours) Target-Calibrated	Original SAM3 Concept Drift Failure

2. Data Drift: Domain Adaptation

Scenario: Applying the model to iSAID (Remote Sensing), a domain with significant visual distribution shifts compared to natural scenes. Observation:

• Original SAM3: Misses objects and hallucinates incorrect labels due to Data Drift.
• ConceptBank (Ours): Anchors to visual prototypes from the support set, successfully segmenting all planes without false positives.

Input Image	ConceptBank (Ours) Target-Calibrated	Original SAM3 Data Drift Failure

---

I. Abstract

Open-vocabulary segmentation (OVS) specifies categories through language, but text-conditioned states can become unreliable under target-domain shift. We study an offline support-assisted setting in which a language-specified deployment vocabulary and a disjoint target-domain support split covering that vocabulary are available once before inference. This protocol is not support-free or zero-shot; rather, it asks whether a frozen language-conditioned segmenter can be calibrated without optimizing model, adapter, or prompt parameters and without retrieving supports at test time. We attribute prompt-target misalignment to data drift, which changes class appearance, and taxonomy-induced concept shift, which changes the operational meaning of a label. We propose ConceptBank, which estimates target visual prototypes, mines prototype-consistent supports, functionally scores candidate descriptions by their induced masks, and fuses the strongest structured conditioning states into one anchor per supported class. Across eight natural-scene evaluation protocols and four remote-sensing datasets, pseudo-mask-supported ConceptBank improves frozen SAM3 from 57.5 to 64.5 and from 39.1 to 47.5 average mIoU; using ground-truth support masks further raises performance to 67.1 and 52.1. The resulting bank preserves cached single-prompt throughput and requires neither support retrieval nor parameter updates at inference.

---

II. Environment Preparation

This project relies on PyTorch, MMSegmentation, and the SAM3 codebase.

Requirements

• Python: 3.12 (Recommended)
• CUDA: 12.6

We strictly tested the code with the following library versions:

• torch 2.9.1+cu126
• torchvision 0.24.1+cu126
• openmim 0.3.9
• mmengine 0.10.7
• mmcv 2.2.0 (Must be built from source)
• mmsegmentation 1.2.2

Installation Steps

1. Create a virtual environment:

conda create -n cb4ovs python=3.12 -y
conda activate cb4ovs

2. Install PyTorch:

pip install torch==2.9.1+cu126 torchvision==0.24.1+cu126 --extra-index-url https://download.pytorch.org/whl/cu126

3. Install MMLab Dependencies: Install basic tools:

pip install openmim==0.3.9
mim install mmengine==0.10.7

Install MMCV (Build from Source): Since we use a specific PyTorch version, mmcv must be compiled from source to ensure CUDA operator compatibility.

For more details, refer to the MMCV Build Guide.

git clone https://github.com/open-mmlab/mmcv.git
cd mmcv
git checkout v2.2.0
# MMCV_WITH_OPS=1 ensures CUDA ops are compiled
MMCV_WITH_OPS=1 pip install -e .
cd ..

Install MMSegmentation:

mim install mmsegmentation==1.2.2

---

III. Setup SAM3 & Checkpoints

Our method is based on SAM3. You must download the official SAM3 repository and weights.

1. Clone SAM3: Download the SAM3 code and place the sam3 folder in the root of this project.

# Ensure the folder is named 'sam3' and sits next to our scripts
git clone https://github.com/facebookresearch/sam3.git

2. Download Checkpoints: Download sam3.pt from the Hugging Face repository. Move the checkpoint to sam3/assets/.

# Move your downloaded sam3.pt here
mv path/to/sam3.pt sam3/assets/

---

IV. Dataset Preparation

Please organize your datasets in a data/ directory.

Natural Scene Datasets

We evaluate on 8 benchmarks. Please follow the MMSegmentation Dataset Preparation Guide for standard datasets:

• MMSegmentation Dataset Prepare

Note on COCO-Object: For the COCO-Object dataset conversion, we follow the format used in SCLIP. Please refer to their repository for the conversion scripts:

• SCLIP GitHub

Remote Sensing Datasets

We evaluate on 4 benchmarks: LoveDA, Potsdam, Vaihingen, and iSAID. Ensure these are formatted compatibly with MMSegmentation conventions.

---

V. Usage

We provide pre-constructed Concept Banks to facilitate direct reproduction of our results. These files are located in:

• configs/concept_bank/cb_sam3_ns.pt (for Natural Scene) download
• configs/concept_bank/cb_sam3_rs.pt (for Remote Sensing) download

You can use the provided scripts to evaluate these banks directly or regenerate them from scratch.

Evaluation (Default)

By default, the scripts cb4ovs_ns.sh and cb4ovs_rs.sh are configured to load the provided Concept Banks and run the evaluation immediately.

Natural Scene Evaluation:

# Usage: bash cb4ovs_ns.sh [NGPUS] [LOGFILE]
bash cb4ovs_ns.sh 4 logs_ns.txt

Remote Sensing Evaluation:

# Usage: bash cb4ovs_rs.sh [NGPUS] [LOGFILE]
bash cb4ovs_rs.sh 4 logs_rs.txt

Concept Bank Generation (Optional)

If you wish to re-build the Concept Banks from scratch (e.g., to reproduce the generation process), simply uncomment the concept generation section (the sam3_concept_bank.py command) in the corresponding shell scripts (cb4ovs_ns.sh or cb4ovs_rs.sh) before running them.

Gradio Demo

We provide a web-based demo for interactive testing.

Install gradio:

pip install gradio==6.2.0

Concept Bank Demo:

python app.py

---

VI. Results

Quantitative Results

We provide the quantitative comparison with state-of-the-art methods on both natural scene and remote sensing benchmarks.

Natural Scene Benchmarks

Remote Sensing Benchmarks

Qualitative Visualization

Natural Scene Visualization

Remote Sensing Visualization

---

VII. Acknowledgements

We explicitly thank the authors of the following excellent open-source projects, which were instrumental in this work:

• SAM3: https://github.com/facebookresearch/sam3
• SCLIP: https://github.com/wangf3014/SCLIP
• TCL: https://github.com/kakaobrain/tcl
• GroupViT: https://github.com/NVlabs/GroupViT
• SegEarth-OV: https://github.com/likyoo/SegEarth-OV