fix(lambda): report the real framework version, and unblock the ECR scans - #6764
Open
Eren-Jeager123 wants to merge 4 commits into
Open
Eren-Jeager123 wants to merge 4 commits into
Eren-Jeager123 wants to merge 4 commits into
Conversation
…mages
pytorch configs claimed 2.11.0 while pyproject pins torch==2.13.0, and the
vllm configs carried the CUDA version (13.0.3) instead of vllm 0.27.1. The
value is baked into bash_telemetry.sh at build time, so usage metrics for
these four images were labelled with the wrong version.
Release tags are unaffected: the lambda entry in DLContainersReleaseLogicPython
frameworks.yml renders release_tag as "{container_type}" and never references
{version}, and its version_pattern is '.*'.
Signed-off-by: Kevin Wang <kwanggg@amazon.com>
Eren-Jeager123
marked this pull request as ready for review
September 18, 2026 18:55
sirutBuasai
approved these changes
Sep 18, 2026
RUSTSEC-2026-0195 (quick-xml DoS in the bundled uv binary) hit its 2026-09-10 review date, failing every lambda ecr-vulnerability-scan job. Bump to 2026-12-31, matching the same entry on main (#6738).
Path traversal via save_pretrained() in transformers <=5.8.0.dev0, fixed in 5.10.0. 5.10.0 itself is yanked upstream, so pin 5.10.4.
RCE in the LlavaOnevision2 processor loader (ignores trust_remote_code), fixed in 0.28.0; take 0.29.0 as current. flashinfer moves to 0.6.18 to match 0.29.0's requirements/cuda.txt. The torch trio pin is unchanged (2.13.0 / 2.11.0 / 0.28.0), so torch-constraints.txt still applies, and the build-system requires are identical between the two refs.
sallyseok
approved these changes
Sep 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
metadata.framework_versionwas wrong on 4 of the 10 Lambda image configs:core/pytorch.yml2.11.02.13.0docker/lambda/pytorch/pyproject.toml:26pinstorch==2.13.0core/pytorch-preview.yml2.11.02.13.0vllm/vllm.yml13.0.30.29.0build.vllm_versionin the same file (was reporting the CUDA version)vllm/vllm-preview.yml13.0.30.29.0The value is baked into
bash_telemetry.shat build time (docker/lambda/Dockerfile:418-425), so usage metrics for these four images carried the wrong version — the vllm images were reporting CUDA13.0.3as their framework version.core/base*.ymlandcore/cupy*.ymlintentionally keep the CUDA version, matching how this repo'sbaseframework does it (.github/config/image/base/cu130-runtime.yml="13.0.2").sglang/*.ymlwas already correct.Also in this PR: unblocking the ECR scans
The
security-test / ecr-vulnerability-scanjob was failing on all 10 Lambda images — a pre-existinglambda-branch breakage, not caused by the metadata change. Three fixes, in the order they were uncovered:1. Expired allowlist review date (
test/security/data/ecr_scan_allowlist/global_allowlist.json)ecr_scan.py:114exits 1 on any past-due entry before it evaluates findings, so every Lambda image failed identically.mainalready refreshed this same entry to2026-12-31in #6738; thelambdabranch hadn't picked it up. Bumped to match. That alone turned base and cupy (4 images) green.2.
CVE-2026-9856— HIGH,exploit=YES— pytorch imagesPath traversal via
save_pretrained()intransformers <= 5.8.0.dev0. Bumpeddocker/lambda/pytorch/pyproject.tomltransformers5.5.0→5.10.4and relocked. The fix first ships in5.10.0, but that release is yanked upstream ("We pushed from a week old main branch […] missing a bunch of fixes"), so5.10.4is the first usable version. The relock touched onlytransformers— no transitive churn.3.
CVE-2026-90553— HIGH — vllm imagesRCE in the
LlavaOnevision2processor loader, which ignorestrust_remote_code. Fixed in0.28.0; taking0.29.0as current. In both vllm configs:vllm_ref6adad087→98dff2a8(tagv0.29.0),vllm_version→0.29.0flashinfer_version0.6.16.post3→0.6.18, matching0.29.0'srequirements/cuda.txtWhat keeps this bump contained, checked against both refs:
torch==2.13.0,torchaudio==2.11.0,torchvision==0.28.0), sodocker/lambda/vllm/torch-constraints.txtstill pins the resolve correctly and torch is not reinstalled.[build-system].requiresinpyproject.tomlis byte-identical between the two refs, includingsetuptools>=77.0.3,<81.0.0andtorch == 2.13.0— so the builder stage's assumptions hold.nvidia-cutlass-dsl|quack-kernels|humming-kernelsfilter (Dockerfile:313) still matches0.29.0'scuda.txt, so the FA4 cute-DSL cluster is still dropped.0.29.0isinstanttensor>=0.1.9, resolved from PyPI by the existing runtime-deps install.0.29.0requirestransformers>=5.10.4, which the pytorch pin in fix 2 now satisfies exactly.Release tags are unaffected
Verified against every consumer of
metadata.framework_version:generate-release-spec/generate_release_spec.sh:30→ release specversion:. Thelambdaentry inDLContainersReleaseLogicPythonsrc/dlc_release_logic/config/frameworks.yml:915-928rendersrelease_tag: "{container_type}"and references{version}in no template;version_patternis'.*'. Prod tags stayvllm-cuda/pytorch-cuda(+-v1,-v1.0,-v1.0.<patch>from the Dockerfiledlc_*_versionLABELs).upload-ecr-allowlists/upload_ecr_allowlists.py:121→ optionallambda/lambda-<version>.json. No such file exists for any version, so the path is skipped either way._reusable.telemetry-tests.yml:134→deep_learning_container.py:281validates the value as PEP 440.2.13.0and0.29.0both parse.build-image/compute_ci_tag.sh:32→ the CI tag changes, so these 4 images rebuild once:lambda-vllm-13.0.3-...→lambda-vllm-0.29.0-gpu-py313-cu130-<suffix>lambda-pytorch-2.11.0-...→lambda-pytorch-2.13.0-gpu-py313-cu130-<suffix>Build cost
Unlike the metadata-only version of this change, the vLLM source compile does re-run:
scripts/ci/build/lambda/lib/source_hash.shkeys the cached wheel onvllm_ref+vllm_version+ arch list, and the first two changed. Expect a cold ~85-min compile per vllm image on this PR and on the first build after merge.Testing
generate_release_spec.shrun against the edited configs —container_typeunchanged, onlyversion:moves.compute_ci_tag.shrun before/after to confirm the exact tag delta above.pre-commit run --files <the 4 configs>— all applicable hooks pass.uv lockondocker/lambda/pytorchregenerates cleanly withtransformers==5.10.4; lock diff is confined to that one package.ecr-vulnerability-scanis the real gate for fixes 1–3 and is still running for the pytorch and vllm images at the time of writing.One knock-on effect worth flagging for whoever looks at dashboards: telemetry for these images splits across the old and new
framework_versionvalues at the cutover. That is the intended outcome of the fix.