SPARC turns robot demonstrations into structured spatial annotations: object boxes, motion traces, manipulation phases, and interaction-based selection scores. It combines task parsing, object detection, segmentation, and tracking to identify the objects a robot actually interacts with.
See the paper for the method and experiments, or explore demonstrations and results on the project page.
This repository contains the annotation pipeline. The project page covers the broader paper, including annotation reliability and downstream learning results.
| Resource | What you'll find |
|---|---|
| SPARC collection | Paper, model weights, datasets and benchmark in one place |
| SPARC-Qwen3.5-4B | Model fine-tuned on SPARC-generated VQA for embodied spatial reasoning |
| SPARC-VQA | Visual question-answering data generated from SPARC annotations |
| IA-Bench | Human-annotated benchmark for grounding interacted objects in robot demonstrations |
The model and dataset cards include usage examples. To generate annotations for your own demonstrations, follow the quickstart below.
SPARC-generated spatial annotations are released per source dataset, with bounding boxes, object traces, manipulation phases and reliability scores.
| Source dataset | Annotation release |
|---|---|
| DROID | 🤗 sparc-droid-annotations |
| BridgeData | Coming soon |
| AgiBotWorld | Coming soon |
| Open X-Embodiment | Coming soon |
Find all releases in the SPARC collection.
You’ll need: Linux, Python 3.12, NVIDIA GPUs, Conda, Git, FFmpeg, and a C/C++ compiler. Run the commands from this repository’s root on a GPU machine; on a managed cluster, use a compute allocation.
conda create -n sparc -c conda-forge python=3.12 pip ffmpeg
conda activate sparc
python -m pip install --no-deps -r requirements.txt
# SAM2 checkpoint
bash sparc/detectors/sam2/checkpoints/download_ckpts.sh
# Download robotseg.pt first, then prepare RobotSeg
python setup/prepare_robotseg.py --checkpoint /path/to/robotseg.ptGet robotseg.pt from the RobotSeg checkpoint links.
LLMDet, AllTracker, and MoGe download weights on first use.
requirements.txt contains the complete pinned annotation environment.
The --no-deps flag is intentional; see
installation notes and CUDA extension builds
for dependency constraints, offline preparation, and validation scope.
Task parsing uses an OpenAI-compatible endpoint, with qwen3-vl-30b as the
served model name. Use an existing endpoint, or start the supplied Qwen server
in a separate environment on a GPU allocated for serving:
conda create -n sparc-vlm python=3.12 pip
conda activate sparc-vlm
python -m pip install -r requirements-vllm.txt
bash slurm/serve_vlm.shKeep the server running. In another shell, activate sparc for annotation.
See VLM setup for model overrides and in-job serving.
Next, follow the workflow below to build a task cache, annotate trajectories, and select annotations by their reliability score.
Demonstrations → task-object cache → spatial annotations → score filtering
SPARC first decomposes instructions into subtasks and identifies task objects.
For each subtask, it detects candidate objects with LLMDet, segments them with
SAM2, and tracks their motion with AllTracker. RobotSeg identifies the robot,
and MoGe supplies 3D geometry for scoring. Phase-aware motion, gripper proximity,
and robot overlap help select the interacted object and its target box.
The resulting interaction-based reliability score is stored as selection.score.
With the VLM server running, precompute the task descriptions:
conda activate sparc
mkdir -p runs/droid
python extract_task_objects.py --dataset droid_lerobot \
dataset.root=/path/to/droid_success \
dataset.task_obj_dict_path=./runs/droid/task_objects.pkl \
vllm.base_url=http://your-vlm-host:8000/v1This writes a reusable task_objects.pkl cache. Use the same dataset, parsing
mode, and cache path in the annotation step. The pre-pass processes the dataset;
extract_task_objects.max_trajectories can limit it. It is optional: annotation
can also resolve and cache missing task descriptions through the VLM server.
Tune extraction with extract_task_objects.n_workers=8 and
extract_task_objects.vlm_concurrency=64; vLLM batches the concurrent requests.
See batching controls.
Point dataset.root at a LeRobot DROID directory containing meta/info.json,
data/, and videos/. Replace the endpoint with your VLM server’s address.
conda activate sparc
mkdir -p runs/droid
python annotate.py --dataset droid_lerobot \
dataset.root=/path/to/droid_success \
dataset.task_obj_dict_path=./runs/droid/task_objects.pkl \
annotator.output_file=./runs/droid/annotations.jsonl \
annotator.gpu_ids=[0] annotator.n_processes_per_gpu=1 \
debug.n_trajectories=2 debug.debug_image_freq=1 \
debug.debug_image_dir=./runs/droid/debug \
vllm.base_url=http://your-vlm-host:8000/v1This uses the left external camera and writes annotations, array shards, and
debug images under runs/droid/. To process the full dataset without debug
images, set debug.n_trajectories=null debug.save_debug_images=false.
Rerunning with the same output path skips existing annotations.
For lower VRAM use, start with annotator.pointcloud_batch_size=8,
annotator.tracking_max_frames=200, and annotator.robotseg_max_frames=32.
See memory tuning for all controls.
Use selection.score, rather than detector confidence, to filter annotations.
Start with a threshold of 0.95, then tune it for your dataset. Higher
thresholds generally trade coverage for quality; 0.95 does not mean 95%
correctness. Check representative examples before choosing an operating point.
Filtering happens per subtask, so a trajectory may retain only some subtasks.
Detector confidence is the detector-only baseline score, not SPARC's reliability score:
| Field | Meaning |
|---|---|
selection.score |
SPARC's interaction-based reliability ranking; use this for the filter below |
baselines.detector.initial.detector_score |
Detector confidence for the baseline's highest-confidence initial box |
baselines.detector.target.detector_score |
Detector confidence for the baseline's highest-confidence target box, when available |
object.initial.detector_score |
Detector confidence of the box selected by SPARC; that box can differ from the detector baseline |
Detector confidence and SPARC's score use different criteria; the 0.95
reliability threshold should not be interpreted as a detector-confidence cutoff.
python filter_annotations.py runs/droid/annotations.jsonl --threshold 0.95For bimanual annotations, every arm's selection.score must meet the threshold;
the command keeps or drops the complete annotation.
This creates runs/droid/annotations_filtered.jsonl and reports how many records
were retained. Missing, nonnumeric, and nonfinite scores are skipped. Use
--output /path/to/filtered.jsonl to choose another location; relative array
references are adjusted automatically. Existing files are never overwritten.
Filtering leaves the source JSONL and shared array shards intact. When sharing
the filtered output, include its referenced annotations_artifacts/ directory.
More examples: right-camera DROID · LIBERO · multi-node SLURM · all configuration options
Each subtask produces one JSONL annotation record. Coordinates refer to the original image; frame windows are start-inclusive and end-exclusive.
| Output | Contents |
|---|---|
annotations.jsonl |
Task, phases, selected boxes, scores, source identity, and array references |
annotations_artifacts/ |
HDF5 shards containing masks and tracks; point clouds are opt-in |
debug/ |
Visual annotation checks and JSON metadata |
tracks/ (optional) |
Raw diagnostic exports with annotator.save_tracks_npz=true |
Point-cloud saving is off by default. Set annotator.save_pointclouds=true
to save object/gripper keyframe clouds and include 3D tracks in optional NPZ
exports. This requires annotator.enable_pointcloud=true (the default).
Saving is separate from computation: MoGe still supplies 3D scoring signals
when save_pointclouds=false. To disable MoGe itself, use
annotator.enable_pointcloud=false, which also changes the available scoring signals.
Keep the array-shard directory alongside the JSONL when moving outputs. Load arrays with the same helper for HDF5, NPZ sidecars, or embedded storage:
import json
from pathlib import Path
from sparc.pipeline.publication_schema import load_publication_arrays
path = Path("runs/droid/annotations.jsonl")
with path.open() as stream:
annotation = json.loads(next(stream))
arrays = load_publication_arrays(annotation, sidecar_root=path.parent)Selection scores rank annotations; they are not calibrated probabilities. See the output reference for fields, coordinate conventions, bimanual annotations, and storage options.
| Set up checkpoints or CUDA extensions | Installation |
| Annotate another dataset | Supported datasets |
| Tune extraction batching or GPU memory | Performance tuning |
| Change GPUs, workers, parsing, or debug settings | Configuration |
| Use a starVLA dataset mixture | starVLA integration |
| Launch, resume, or merge a multi-node run | SLURM guide |
| Understand the implementation | How it works · Repository layout |
| Run the tests | Testing |
If you use SPARC, please cite our paper.
@misc{blank2026sparc,
title = {{SPARC}: Reliable Spatial Annotations from Robot Demonstrations at Scale},
author = {Nils Blank and Paul Mattes and Maximilian Xiling Li and Jakub Suliga and Thomas Roth and Moritz Reuss and Pankhuri Vanjani and Rudolf Lioutikov},
year = {2026},
eprint = {2606.13497},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2606.13497}
}SPARC builds on LLMDet, AllTracker, SAM2, RobotSeg, and MoGe. We thank their authors for making these models available. See vendor scope for the inference components included here and their upstream licenses.