Skip to content

feat(abot): add multi-session ABot-World serving and observability - #31

Open
youngmagician114514 wants to merge 8 commits into
Tele-AI:mainfrom
youngmagician114514:feat/abot-world-turboserve-serving
Open

feat(abot): add multi-session ABot-World serving and observability#31
youngmagician114514 wants to merge 8 commits into
Tele-AI:mainfrom
youngmagician114514:feat/abot-world-turboserve-serving

Conversation

@youngmagician114514

Copy link
Copy Markdown
Contributor

Description

Adds ABot-World multi-session interactive serving with bounded process and
process-NCCL workers, deadline-aware continuation batching, opt-in CUDA Graph
execution, and reproducible serving observability tooling.

Motivation

ABot-World previously lacked an end-to-end runtime for multiple concurrent
interactive sessions. This PR adds explicit session ownership, lifecycle
cleanup, capacity enforcement, frame-credit-aware EDF scheduling, and raw
runtime facts for reproducible evaluation.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Performance improvement
  • Code refactoring
  • Documentation update
  • Other

Changes Made

  • Add multi-session ABot-World serving with process and process-nccl.
  • Add session migration, bounded capacity, and EOS cleanup.
  • Add frame-credit-aware EDF B1/B2/B3 batching and batch-time profiles.
  • Add opt-in CUDA Graph with eager fallback and parity validation.
  • Add trace replay, metrics, analysis tools, and monitoring configuration.

Testing

  • Focused unit tests pass
  • Manual multi-GPU serving validation performed
  • Benchmarks and trace-validation tooling added/updated

Test commands:

PYTHONPATH=$PWD /public/fanyk1/lwb/envs/telefuser_sage291/bin/python -m pytest -q \
  tests/unit/pipelines/abot_world \
  tests/unit/service/livekit \
  tests/unit/validation

ruff check <changed-python-files>
ruff format --check <changed-python-files>
git diff --check
bash -n tools/validation/run_abot_4gpu_30min_trace.sh

Focused result: 138 passed.

Checklist

  • Code follows the project's coding standards (ruff)
  • Pre-commit hooks pass (pre-commit run --all-files)
  • All tests pass (pytest tests/)
  • New tests added for new functionality
  • Documentation updated
  • Commit messages are clear and descriptive
  • PR title follows the convention: [Feature] Brief description

Related Issues

None.

Additional Notes

  • CUDA Graph is opt-in and disabled by default.
  • EDF batching is opt-in and preserves conservative default behavior.
  • The supplied TurboServe trace is a public demo workload, not a released
    private production trace.
  • Local results/, .orig, archives, and raw experiment files are excluded.

GPU Architecture Support

  • SM80 (Ampere, Ada Lovelace)
  • SM90 (Hopper H100)
  • SM100+ (Blackwell)

Performance Impact

The batching and CUDA Graph paths are opt-in and retain eager fallback.
Batch-time priors are profile-specific; this PR does not claim a universal
throughput improvement.

Handle minimal CPU pipeline stubs without requiring CUDA Graph methods.
Validate CUDA Graph environment values explicitly and document the attention
and CUDA Graph opt-in controls.

Verification:
- 138 focused ABot serving, scheduler, NCCL, and trace tests passed
- branch Ruff and whitespace checks passed
- four-GPU trace runner syntax check passed
Apply Ruff formatting and import ordering to the ABot serving,
NCCL, scheduler, and validation changes.

Verification:
- Ruff check and format check passed
- git diff --check passed
Use tracked workload assets, keep upstream trace regeneration optional, and register the missing service example.
@lzx1413
lzx1413 requested a review from Kaimary August 19, 2026 13:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant