Skip to content

feat(manipulation): add live grasp visualization - #3365

Draft
TomCC7 wants to merge 4 commits into
feat/grasp-03-connected-plansfrom
feat/grasp-04-visualization
Draft

feat(manipulation): add live grasp visualization#3365
TomCC7 wants to merge 4 commits into
feat/grasp-03-connected-plansfrom
feat/grasp-04-visualization

Conversation

@TomCC7

@TomCC7 TomCC7 commented Aug 5, 2026

Copy link
Copy Markdown
Member

Contribution path

Problem

Operators cannot inspect ranked grasp candidates, the selected grasp, its approach path, or execution state through the manipulation visualizer. This makes planning failures and unexpected selections difficult to diagnose.

Solution

  • Add reusable manipulation visualization layers.
  • Render ranked candidates, the selected grasp, approach paths, and live execution state.
  • Add Viser integration and hermetic tests.
  • Keep production grasp geometry self-contained instead of importing demo code.

This is stack 4 of 5. Previous: #3364. Next: #3366. Review the diff against feat/grasp-03-connected-plans.

How to Test

uv run pytest dimos/manipulation/visualization dimos/manipulation/planning/monitor/test_world_monitor.py -v

Verified: 168 generic visualization and world-monitor tests passed; 78 focused live-grasp visualization and pick tests passed; Ruff passed.

AI assistance

OpenAI Codex with GPT-5 was used extensively for branch extraction, implementation cleanup, verification, and this description.

Checklist

  • I have read and approved the CLA.

@codecov

codecov Bot commented Aug 5, 2026

Copy link
Copy Markdown

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
3625 1 3624 175
View the top 1 failed test(s) by shortest run time
dimos.robot.test_all_blueprints_generation::test_all_blueprints_is_current
Stack Traces | 4.1s run time
def test_all_blueprints_is_current() -> None:
        root = DIMOS_PROJECT_ROOT / "dimos"
        all_blueprints, all_modules = _scan_for_blueprints(root)
    
        common = set(all_blueprints.keys()) & set(all_modules.keys())
        assert not common, (
            f"Names must be unique across blueprints and modules, "
            f"but these appear in both: {sorted(common)}"
        )
    
        generated_content = _generate_all_blueprints_content(all_blueprints, all_modules)
    
        file_path = root / "robot" / "all_blueprints.py"
    
        if "CI" in os.environ:
            if not file_path.exists():
                pytest.fail(f"all_blueprints.py does not exist at {file_path}")
    
            current_content = file_path.read_text()
            if current_content != generated_content:
                diff = difflib.unified_diff(
                    current_content.splitlines(keepends=True),
                    generated_content.splitlines(keepends=True),
                    fromfile="all_blueprints.py (current)",
                    tofile="all_blueprints.py (generated)",
                )
                diff_str = "".join(diff)
>               pytest.fail(
                    f"all_blueprints.py is out of date. Run "
                    f"`pytest dimos/robot/test_all_blueprints_generation.py` locally to update.\n\n"
                    f"Diff:\n{diff_str}"
                )
E               Failed: all_blueprints.py is out of date. Run `pytest dimos/robot/test_all_blueprints_generation.py` locally to update.
E               
E               Diff:
E               --- all_blueprints.py (current)
E               +++ all_blueprints.py (generated)
E               @@ -216,6 +216,7 @@
E                    "goal-relay": "dimos.navigation.nav_3d.mls_planner.goal_relay.GoalRelay",
E                    "google-maps-skill-container": "dimos.agents.skills.google_maps_skill_container.GoogleMapsSkillContainer",
E                    "gps-nav-skill-container": "dimos.agents.skills.gps_nav_skill.GpsNavSkillContainer",
E               +    "grasp-gen-x-module": "dimos.manipulation.grasping.grasp_gen_x.GraspGenXModule",
E                    "grasping-module": "dimos.manipulation.grasping.grasping.GraspingModule",
E                    "gstreamer-camera-module": "dimos.hardware.sensors.camera.gstreamer.gstreamer_camera.GstreamerCameraModule",
E                    "hosted-stats-module": "dimos.teleop.hosted.hosted_stats.HostedStatsModule",

all_blueprints = {'a1z-planner-coordinator': 'dimos.robot.manipulators.a1z.blueprints.basic:a1z_planner_coordinator', 'alfred-nav': 'di...rs.a1z.blueprints.basic:coordinator_a1z', 'coordinator-basic': 'dimos.control.blueprints.basic:coordinator_basic', ...}
all_modules = {'alfred-high-level': 'dimos.robot.diy.alfred.effector_high_level.AlfredHighLevel', 'arm-command-module': 'dimos.teleo...t_extensions.ArmTeleopModule', 'b-box-navigation-module': 'dimos.navigation.bbox_navigation.BBoxNavigationModule', ...}
common     = set()
current_content = '# Copyright 2025-2026 Dimensional Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the "License");\n# you m...sted.blueprints.cloudflare.WristCamera",\n    "zed-camera": "dimos.hardware.sensors.camera.zed.camera.ZEDCamera",\n}\n'
diff       = <generator object unified_diff at 0xff94f4bd86c0>
diff_str   = '--- all_blueprints.py (current)\n+++ all_blueprints.py (generated)\n@@ -216,6 +216,7 @@\n     "goal-relay": "dimos.na...er_camera.GstreamerCameraModule",\n     "hosted-stats-module": "dimos.teleop.hosted.hosted_stats.HostedStatsModule",\n'
file_path  = PosixPath('.../dimos/robot/all_blueprints.py')
generated_content = '# Copyright 2025-2026 Dimensional Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the "License");\n# you m...sted.blueprints.cloudflare.WristCamera",\n    "zed-camera": "dimos.hardware.sensors.camera.zed.camera.ZEDCamera",\n}\n'
root       = PosixPath('.../dimos/dimos/dimos')

dimos/robot/test_all_blueprints_generation.py:76: Failed

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

TomCC7 and others added 3 commits August 4, 2026 22:42
Publish the object cloud, the ranked grasp candidates, and the candidate
currently under feasibility check as Viser display layers, so a failing
pick can be inspected instead of inferred from logs.

The gripper geometry comes from config rather than the generator: grasp_gen_x
bakes grasp_frame_to_tcp into candidate poses and the wireframe un-applies it,
so the xarm blueprint sources both from one config object. Publishing is
opt-in, and every call is suppressed so visualization cannot fail a pick.
@TomCC7
TomCC7 force-pushed the feat/grasp-04-visualization branch from 74e7634 to 4cfb213 Compare August 5, 2026 05:45
@TomCC7
TomCC7 force-pushed the feat/grasp-03-connected-plans branch from 9dcfb82 to 1cd98d4 Compare August 5, 2026 05:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants