Add SAM3 perception and persistent placed-box obstacles to hangar_sim ML Move Boxes

[griswaldbrooks]

[written by AI]

Problem

Two issues in hangar_sim's ML Move Boxes to Loading Zone: (1) perception used the CLIPSeg+SAM2 pipeline with an unbounded RetryUntilSuccessful(1000) loop that never terminated and hid real errors (moveit_pro#19112); (2) already-placed boxes were invisible to planning, so the base could drive over them and the arm could sweep through them (part of moveit_pro#19973).

Full objective run, sped up ~13× (7.6 min real time → 35 s) — every scattered box picked off the hangar floor and placed in the loading zone, objective terminating with SUCCESS via the new drain detection. Higher-quality 720p: pr746_full_run.mp4 (1.6 MB).

Changes

• SAM3 perception: migrate to GetMasks2DFromExemplar (single text-prompted segmenter, replaces CLIPSeg+SAM2 pair), add FilterMasks2DByArea, and restructure the loop to be drain-aware: RepeatUnlessFailureEachTick + a drained script flag, so the objective terminates SUCCESS when the scene is empty and FAILURE on real pick errors instead of retrying forever. Includes a calibrate_sam3_mask_areas diagnostic objective.
• Persistent placed-box obstacles: each placed box becomes its own world collision object (placed_box_N, AddCollisionMesh with overwrite), pose dropped to floor height via a body-frame TransformPose; ResetPlanningSceneObjects at objective start clears stale objects from prior runs. Subsequent base/arm plans route around placed boxes.
• Grasp-pose leveling: every candidate grasp's orientation is overridden to straight-down (ForEach → OverridePoseOrientation → rebuilt vector) before MTC planning, discarding noisy segmentation yaw. Note this makes the wrist-orientation samples duplicates of each other (see tuning below).
• Post-grasp lift + pick tuning: 10 cm Cartesian lift along grasp_link −Z (world up) before transit so the box clears the surface; num_orientation_samples 20→8 and max_ik_solutions 16→8 (pose IK is the cycle-time bottleneck; with leveling the orientation samples collapse anyway — a follow-up may reduce to 1).

Behavior-tree renderings (MoveIt Pro editor)

Top-level Objective — drain-aware loop: init scripts, then Fallback[RepeatUnlessFailureEachTick → ScriptCondition(drained)] alternating the two view-waypoint SubTree calls (second gated by _skipIf: picked_at_wp1):

Pick subtree — full shape at a glance, then readable sections left to right:

Section	Rendering
Capture + SAM3 perception + 3D extraction
Grasp generation + MTC task assembly
Lift + place + placed-box obstacles

Annotated structure diagrams (green = new in this PR vs. gray = pre-existing, derived from the diff — shows what changed structurally, which the editor view can't)

Structural changes worth a reviewer's attention: the pick-resilience retry loop moved from iterating 2D masks (with per-mask SAM2 refinement, now removed) to iterating post-3D-extraction graspable_objects; the _skipIf: mask_count == 0 on the outer Inverter is what lets a "scanned, found nothing" pass exit SUCCESS with picked_box=false, which the parent's drain logic depends on.

_{Images live on the assets/pr746-bt-diagrams branch. UI renderings captured from the live editor; annotated diagrams generated programmatically from the XML.}

Scope note (2026-07-02 rescope)

An earlier revision of this PR carried cost_joint_names/cost_joint_weights yaw-bias ports that depended on core PR moveit_pro#20073; that PR and example_ws#736 are closed and the ports are removed — this PR is standalone-mergeable on current main. The base-yaw pirouette those ports partially mitigated is fixed properly elsewhere: controller-level mitigation in #753 (angle_wraparound on the yaw JTC gain, validated 14 runs / 0 spins) and the core representation-splice fix tracked in moveit_pro#20250.

Verification

• Heavily exercised live on 2026-07-02 (this objective content was the test vehicle for the Fix: Stop hangar_sim base pirouette by enabling JTC yaw angle_wraparound #753 validation campaign): 11 reset-controlled + 3 human-driven runs — SAM3 picking 7/7 boxes on successful runs, drain-detection terminating with objective SUCCESS in ~7.5 min, placed_box_* obstacles present in the planning scene, placed boxes landing in a clean line in the loading zone.
• Post-rescope retest of the exact merged content (ports removed, rebased on main): end-to-end run against a live backend — terminal SUCCESS in ~8.4 min, 7 distinct box welds, spin detector CLEAN (599.9 s joint recording), zero ERROR lines, placed_box_0..7 world collision objects verified via /get_planning_scene. (Object count was 8 rather than the expected 7, most likely carryover from a preceding hung attempt on the same backend session — the persistence mechanism itself is working; noting for transparency.)
• Known unrelated flake: the objective can hang silently mid-move (moveit_pro#20238, pre-existing, ~half of runs in stress testing) — orthogonal to this PR's changes.

Release notes

• Enhancement: Migrated the hangar_sim ML Move Boxes to Loading Zone Objective to SAM3 perception with drain-aware looping, so it finishes when the boxes are gone and reports real failures instead of retrying forever.
• Enhancement: Placed boxes in hangar_sim are now added to the planning scene as obstacles, so the robot plans around boxes it has already placed.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR adds a SAM3 mask calibration tree, changes the top-level loading loop to use drained-state control, and updates the waypoint subtree to use SAM3 exemplar segmentation, revised ports, forced-down grasp orientation, JTC lift execution, and placement-state tracking.

Changes

Behavior tree updates

Layer / File(s)	Summary
SAM3 mask calibration tree src/hangar_sim/objectives/calibrate_sam3_mask_areas.xml, src/hangar_sim/test/objectives_integration_test.py	New behavior tree captures a wrist-camera image, runs SAM3 mask inference, extracts per-mask properties, publishes labeled mask visualizations, logs per-mask details, and includes TreeNodesModel metadata; the integration test skips this objective in CI.
Top-level pick-until-drained loop src/hangar_sim/objectives/ml_move_boxes_to_loading_zone.xml	Replaces retry/fallback control flow with a repeating loop that resets planning-scene objects, tracks per-waypoint pick flags, computes drained, and uses it to gate loop continuation and final success.
Waypoint SAM3 segmentation and ports src/hangar_sim/objectives/move_boxes_to_loading_zone_from_waypoint.xml	Updates the subtree interface to the SAM3 text-prompt contract, resets picked_box on entry, replaces the earlier segmentation chain with GetMasks2DFromExemplar, filters masks by area, publishes mask visualizations, and skips pickup when no masks are detected.
Waypoint grasp and placement flow src/hangar_sim/objectives/move_boxes_to_loading_zone_from_waypoint.xml	Forces grasp orientation to a down-facing quaternion, updates MTC IK settings, executes the lift through JTC, adds a persistent collision mesh at the placed pose, increments placed_count, and marks picked_box true after placement.

Possibly related issues

• PickNikRobotics/moveit_pro#20239 — The changes rework the same ml_move_boxes_to_loading_zone.xml pick-planning flow referenced by the issue.

Possibly related PRs

• PickNikRobotics/moveit_pro_example_ws#692 — It introduced or exercised the objective IDs and skip rules that this PR extends with the new calibration objective.
• PickNikRobotics/moveit_pro_example_ws#732 — It touches the same waypoint grasp-generation path updated here.

Suggested reviewers: bkanator, JWhitleyWork

---

Caution

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

• Ignore

❌ Failed checks (1 error)

Check name	Status	Explanation	Resolution
Human Review Check	❌ Error	FAIL: this PR makes broad hangar_sim objective changes across perception, grasping, placement, planning-scene persistence, and CI skips—too large/cross-cutting for low-risk.	This PR requires review by a requested human reviewer. After review, a non-author requested reviewer should override this pre-merge check.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The description clearly matches the PR changes, covering SAM3 perception, drain-aware looping, and persistent placed-box obstacles.

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

MoveIt Pro Example WS - Objectives Integration Test Report

• lab_sim
  • ✅ jazzy: view HTML report — 35 passed
• hangar_sim
  • ❌ jazzy: view HTML report — 1 failed, 16 passed

[github-actions]

MoveIt Pro Example WS - Objectives Integration Test Report

• lab_sim
  • ✅ jazzy: view HTML report — 35 passed
• hangar_sim
  • ❌ jazzy: view HTML report — 1 failed, 16 passed

[github-actions]

MoveIt Pro Example WS - Objectives Integration Test Report

• lab_sim
  • ✅ jazzy: view HTML report — 35 passed
• hangar_sim
  • ❌ jazzy: view HTML report — 1 failed, 16 passed

[github-actions]

MoveIt Pro Example WS - Objectives Integration Test Report

• lab_sim
  • ✅ jazzy: view HTML report — 35 passed
• hangar_sim
  • ❌ jazzy: view HTML report — 1 failed, 16 passed

[github-actions]

MoveIt Pro Example WS - Objectives Integration Test Report

• lab_sim
  • ✅ jazzy: view HTML report — 35 passed
• hangar_sim
  • ❌ jazzy: view HTML report — 1 failed, 16 passed

[github-actions]

MoveIt Pro Example WS - Objectives Integration Test Report

• lab_sim
  • ✅ jazzy: view HTML report — 35 passed
• hangar_sim
  • ❌ jazzy: view HTML report — 1 failed, 16 passed

[github-actions]

MoveIt Pro Example WS - Objectives Integration Test Report

• lab_sim
  • ✅ jazzy: view HTML report — 35 passed
• hangar_sim
  • ❌ jazzy: view HTML report — 1 failed, 16 passed

[github-actions]

MoveIt Pro Example WS - Objectives Integration Test Report

• lab_sim
  • ✅ jazzy: view HTML report — 35 passed
• hangar_sim
  • ✅ jazzy: view HTML report — 16 passed

[github-actions]

MoveIt Pro Example WS - Objectives Integration Test Report

• lab_sim
  • ✅ jazzy: view HTML report — 35 passed
• hangar_sim
  • ✅ jazzy: view HTML report — 16 passed

[griswaldbrooks]

[written by AI]

@marioprats @JWhitleyWork — review requested from you both, but either one's approval is sufficient; whoever gets to it first. The description carries the full story: a sped-up video of the complete objective run, behavior-tree renderings, and the live retest evidence.

[marioprats]

Looks good to me! nice improvement.
I haven't tested it myself, from the PR description it looks like it has been thoroughly tested already