feat(vllm): arm64/Graviton CPU vLLM DLC (EC2 + SageMaker, AL2023) - #6732
Open
Jyothirmaikottu wants to merge 6 commits into
Open
Jyothirmaikottu wants to merge 6 commits into
Jyothirmaikottu wants to merge 6 commits into
Conversation
Jyothirmaikottu
force-pushed
the
vllm-cpu-arm64-amzn2023
branch
2 times, most recently
from
September 14, 2026 07:01
14c91b8 to
86e31aa
Compare
Jyothirmaikottu
force-pushed
the
vllm-cpu-arm64-amzn2023
branch
from
September 16, 2026 23:27
86e31aa to
68bcf88
Compare
Adds arm64/Graviton CPU vLLM images (EC2 + SageMaker) on AL2023, built from the shared multi-arch docker/vllm/Dockerfile.cpu.amzn2023. Includes arch-keyed build fleet selection, device-type gating for CPU vs GPU test suites, per-matrix SageMaker endpoint tests, and arm64 CPU model/endpoint test entries. Standalone off main; no x86 CPU image config or tests.
Jyothirmaikottu
force-pushed
the
vllm-cpu-arm64-amzn2023
branch
from
September 16, 2026 23:33
68bcf88 to
cf26291
Compare
grpc v1.82.1 (xds.NewGRPCServer) statically linked in mooncake libetcd_wrapper.so; fix only in a grpc 1.85.0-dev pseudo-version, not patchable without a mooncake rebuild. Already allowlisted in the vllm_server allowlist; mirror into the ubuntu vllm allowlist.
The 2B qwen3.5 tarball is a Mamba/GDN hybrid whose GDN kernels are Triton-only (no CPU backend), so the endpoint fails to start within the SageMaker health-check window on Graviton. The dense 0.8B (already used by the EC2 CPU smoke test) starts cleanly, matching the x86 CPU endpoint.
LlavaOnevision2 processor loader RCE, fixed upstream in vllm 0.28.0. AL2023 GPU image is on source-built vllm 0.27.1; short review_by to force the currency bump as the real fix.
Same grpc-in-mooncake CVE already mirrored into the ubuntu vllm allowlist; the AL2023 GPU images (vllm_server) need it too.
sallyseok
requested changes
Sep 18, 2026
| model-tests: | ||
| if: ${{ always() && !cancelled() && inputs.run-model-test && needs.build.result != 'failure' && fromJSON(needs.check.outputs.skips || '{}')['vllm/model'] != true }} | ||
| needs: [build, check] | ||
| if: ${{ always() && !cancelled() && inputs.run-model-test && (needs.ci-config.outputs.device-type == 'gpu' || needs.ci-config.outputs.device-type == 'cpu') && needs.build.result != 'failure' && fromJSON(needs.check.outputs.skips || '{}')['vllm/model'] != true }} |
Collaborator
There was a problem hiding this comment.
Should delete || needs.ci-config.outputs.device-type == 'cpu')
| - ".github/workflows/vllm.pr-amzn2023.yml" | ||
| - ".github/workflows/vllm.tests-unit.yml" | ||
| - "docker/vllm/Dockerfile.amzn2023" | ||
| - "docker/vllm/Dockerfile.cpu.amzn2023" |
Collaborator
There was a problem hiding this comment.
Let's create a separate pr testing workflow file for the cpu Dockerfile. Again, to avoid the situation where a change to one Dockerfile triggers build and tests for the other, unrelated Dockerfile. We can call it vllm.pr-cpu-amzn2023.yml and include both the x86 and arm64 in there, or we can separate further into arm64 cpu and x86 cpu if they differ enough in their tests.
In that file, the discover-configs job would need a different pattern like
- id: discover
uses: ./.github/actions/discover-configs
with:
pattern: ".github/config/image/vllm/*-arm64-amzn2023.yml"
| public_registry: true | ||
| private_registry: true | ||
| enable_soci: true | ||
| environment: "production" |
Collaborator
There was a problem hiding this comment.
This is a new image type, so we should gamma test the release first. Let's set this to environment: "gamma"
| # Smoke tests: model serving + inference validation. | ||
| # smoke-test has codebuild-fleet and runner-scale-sets sub-keys. | ||
| # Workflow parsers construct s3_path from s3_prefix + s3_model. | ||
| # required_image_pattern scopes each entry to gpu vs cpu images. |
Collaborator
There was a problem hiding this comment.
For this file and .github/config/model-tests/vllm-sagemaker-endpoint-tests.yml, we should just create new config files for cpu. I want to err towards separating the test triggering file paths so that small changes don't trigger tests for unrelated images
|
|
||
| ARG PYTHON="python3" | ||
| ARG PYTHON_VERSION=3.12 | ||
| ARG DLC_MAJOR_VERSION=2 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds an arm64 / Graviton CPU vLLM DLC (EC2 + SageMaker, AL2023), the aarch64 companion to the x86 CPU vLLM DLC in #6721. This branch stacks on #6721 — until that merges, the diff here includes the x86 CPU changes too; review only the arm64-specific delta below.
The CPU Dockerfile (
docker/vllm/Dockerfile.cpu.amzn2023, from #6721) is already arch-clean — multi-arch AL2023 base, gcc14, and aVLLM_CPU_ARM_BF16build arg — so arm64 needs config + pipeline wiring only, no new Dockerfile.Changes (arm64 delta)
ec2-arm64-amzn2023.yml,sagemaker-arm64-amzn2023.yml—arch_type: arm64,vllm_cpu_arm_bf16: "1", reusing the CPU Dockerfile and its targets (arch is chosen by the build fleet, not the stage name). Auto-discovered by the PR config glob.vllm.pipeline.yml): arch-key the build fleet —ci-configemitsbuild-fleet(arm64 → arm64-graviton4-build-runner, elsex86-vllm-build-runner); thebuildjob gainsneeds: [ci-config]and reads it. Mirrorsllama-cpp.pipeline.yml(DLC builds native to the runner arch, no--platform).ml.c7g+ml.c8g); the existing x86 CPU entries are scoped witharch_type: x86so they no longer match arm64 images.Verification
+cpuserving proven on a real Graviton4 (c8g.4xlarge, AL2023):/health200 + valid chat completion. Found the decisive gcc14 requirement (AL2023 stock gcc 11.5 can't compile torch-inductor's Armv9 codegen at inference) — already handled in the shared Dockerfile.VLLM_CPU_ARM_BF16is CMake-consumed (cmake/cpu_extension.cmake), aarch64 is a first-class CPU target (armv8.2-a+bf16, FEAT_BF16; present on Graviton 3/4/5), andrequirements/cpu.txtcarries the aarch64torch==2.13.0+cpupin (served via the Dockerfile's PyTorch CPU--extra-index-url).VLLM_CPU_ARM_BF16=1; each image config filters to only its matching test entry (no x86/arm64 cross-matching).Follow-ups (not in this PR)
Kept
release: false— autorelease callers,release: true, ECR CVE-allowlist triage, and a live CPU SageMaker endpoint validation are the remaining release-wiring steps.