Vla test - #32
Open
HappyDog060713 wants to merge 7 commits into
Open
Conversation
…hmark Remove the redundant standalone LingBot VLA v2 FastAPI endpoint while retaining the structured request adapter used by the native TeleFuser service and in-process benchmarks. Replace endpoint-specific tests with focused request validation coverage. Add matched no-capture upstream and TeleFuser runtime benchmark tooling, strict condition comparison, unit tests, reproducible H100 results, and documented interpretation. Verification: 60 focused VLA/native-service tests passed; 153 shared non-LiveKit service tests passed; ruff check and format checks passed; git diff --check passed. Full cross-model service collection was not run in the isolated VLA environment because av, cv2, diffusers, and imageio are intentionally absent.
Document the validated VLA runtime environment and record the baf3d18 native structured HTTP smoke and full-project CI results. Preserve the frozen upstream parity baseline while adding reproducible service evidence. Verified: native 20-request smoke passed with 20/20 finite 50x55 actions; full-project CI passed with 1,236 unit tests and 62 server/API tests.
Register VLA inference time and peak memory as AIPerf record metrics so structured API workloads report model-level latency and accelerator memory alongside request distributions. Add a bounded external sampler for service RSS and per-GPU process memory, and document the isolated workload commands and read-only GPU replica capacity assessment. Keep model parallelism disabled and make the GPU assessment report that one full replica per GPU is the supported mode.\n\nVerification: .venv-aiperf/bin/python -m pytest benchmarks/telefuser_aiperf/tests/test_vla_structured.py -q with the repository adapter on PYTHONPATH (9 passed); ruff check and ruff format --check passed; git diff --cached --check passed.
Document the model layout, feature support, file responsibilities, and VLA-specific execution paths using the structure of the maintained model examples. Keep the strict upstream parity workflow and upstream-versus-TeleFuser runtime performance comparison unchanged.
Add model-scoped TorchAO FP8, tf-kernel FP8, and bitsandbytes NF4 selection for LingBot VLA v2 while preserving BF16 as the default and retaining control-sensitive action and MoE paths in BF16. Expose quantization through the direct example, native structured service, and runtime/service benchmark tools. Add loader, backend selection, dtype compatibility, and service forwarding tests, plus H100 screening results and compatibility guidance. Verification: .venv-vla/bin/python -m ruff check (focused files); .venv-vla/bin/python -m ruff format --check (focused files); .venv-vla/bin/python -m pytest tests/unit/models/test_lingbot_vla_v2_loader.py tests/unit/models/test_lingbot_vla_v2_quantization.py tests/unit/pipelines/lingbot_vla_v2 tests/unit/service/test_structured_tasks.py -q (46 passed). tf-kernel FP8 real-model execution remains pending a matching PyTorch 2.11 CUDA 13.0 SM90 wheel.
Freeze the official 6B quantization manifest and record the effective backend, package, wrapper, and weight identities in VLA artifacts and benchmark reports. Add BF16-to-quantized action comparison, TorchAO/NF4 lifecycle coverage, p99 and shutdown resource metrics, and document quantized loading plus matched upstream inference performance. Verification: 100 VLA-focused unit tests passed; 2 H100 quantization lifecycle tests passed; strict upstream parity passed 38/38; real TorchAO FP8 and BNB NF4 6B action captures completed.
Move the baf3d18 native HTTP and full-project CI evidence into the LingBot VLA v2 example README and remove the standalone validation report. Reorganize the guide around environment, usage, quantization, parity, service validation, performance, and limitations while preserving the strict 38-item upstream comparison and matched upstream runtime benchmark.\n\nVerification:\n- git diff --check\n- verified no remaining VALIDATION_BAF3D18.md references\n- verified balanced Markdown code fences
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Motivation
Type of Change
Changes Made
Testing
pytest tests/)Test commands:
# Add the commands used to test your changes pytest tests/Checklist
ruff)pre-commit run --all-files)pytest tests/)[TYPE] Brief descriptionRelated Issues
Fixes #
Additional Notes
GPU Architecture Support
Performance Impact