Skip to content

fix(lambda): report the real framework version, and unblock the ECR scans - #6764

Open
Eren-Jeager123 wants to merge 4 commits into
lambdafrom
fix/lambda-framework-version-metadata
Open

Eren-Jeager123 wants to merge 4 commits into
lambdafrom
fix/lambda-framework-version-metadata

Conversation

@Eren-Jeager123

@Eren-Jeager123 Eren-Jeager123 commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Problem

metadata.framework_version was wrong on 4 of the 10 Lambda image configs:

Config Was Now Source of truth
core/pytorch.yml 2.11.0 2.13.0 docker/lambda/pytorch/pyproject.toml:26 pins torch==2.13.0
core/pytorch-preview.yml 2.11.0 2.13.0 same
vllm/vllm.yml 13.0.3 0.29.0 build.vllm_version in the same file (was reporting the CUDA version)
vllm/vllm-preview.yml 13.0.3 0.29.0 same

The value is baked into bash_telemetry.sh at build time (docker/lambda/Dockerfile:418-425), so usage metrics for these four images carried the wrong version — the vllm images were reporting CUDA 13.0.3 as their framework version.

core/base*.yml and core/cupy*.yml intentionally keep the CUDA version, matching how this repo's base framework does it (.github/config/image/base/cu130-runtime.yml = "13.0.2"). sglang/*.yml was already correct.

Also in this PR: unblocking the ECR scans

The security-test / ecr-vulnerability-scan job was failing on all 10 Lambda images — a pre-existing lambda-branch breakage, not caused by the metadata change. Three fixes, in the order they were uncovered:

1. Expired allowlist review date (test/security/data/ecr_scan_allowlist/global_allowlist.json)

1 allowlist entries past their review_by date — update or remove them:
  RUSTSEC-2026-0195 (review_by 2026-09-10, test/security/data/ecr_scan_allowlist/global_allowlist.json)

ecr_scan.py:114 exits 1 on any past-due entry before it evaluates findings, so every Lambda image failed identically. main already refreshed this same entry to 2026-12-31 in #6738; the lambda branch hadn't picked it up. Bumped to match. That alone turned base and cupy (4 images) green.

2. CVE-2026-9856 — HIGH, exploit=YES — pytorch images

Path traversal via save_pretrained() in transformers <= 5.8.0.dev0. Bumped docker/lambda/pytorch/pyproject.toml transformers 5.5.05.10.4 and relocked. The fix first ships in 5.10.0, but that release is yanked upstream ("We pushed from a week old main branch […] missing a bunch of fixes"), so 5.10.4 is the first usable version. The relock touched only transformers — no transitive churn.

3. CVE-2026-90553 — HIGH — vllm images

RCE in the LlavaOnevision2 processor loader, which ignores trust_remote_code. Fixed in 0.28.0; taking 0.29.0 as current. In both vllm configs:

  • vllm_ref 6adad08798dff2a8 (tag v0.29.0), vllm_version0.29.0
  • flashinfer_version 0.6.16.post30.6.18, matching 0.29.0's requirements/cuda.txt

What keeps this bump contained, checked against both refs:

  • The torch trio pin is unchanged (torch==2.13.0, torchaudio==2.11.0, torchvision==0.28.0), so docker/lambda/vllm/torch-constraints.txt still pins the resolve correctly and torch is not reinstalled.
  • [build-system].requires in pyproject.toml is byte-identical between the two refs, including setuptools>=77.0.3,<81.0.0 and torch == 2.13.0 — so the builder stage's assumptions hold.
  • The nvidia-cutlass-dsl|quack-kernels|humming-kernels filter (Dockerfile:313) still matches 0.29.0's cuda.txt, so the FA4 cute-DSL cluster is still dropped.
  • New transitive requirement in 0.29.0 is instanttensor>=0.1.9, resolved from PyPI by the existing runtime-deps install.
  • 0.29.0 requires transformers>=5.10.4, which the pytorch pin in fix 2 now satisfies exactly.

Release tags are unaffected

Verified against every consumer of metadata.framework_version:

  • generate-release-spec/generate_release_spec.sh:30 → release spec version:. The lambda entry in DLContainersReleaseLogicPython src/dlc_release_logic/config/frameworks.yml:915-928 renders release_tag: "{container_type}" and references {version} in no template; version_pattern is '.*'. Prod tags stay vllm-cuda / pytorch-cuda (+ -v1, -v1.0, -v1.0.<patch> from the Dockerfile dlc_*_version LABELs).
  • upload-ecr-allowlists/upload_ecr_allowlists.py:121 → optional lambda/lambda-<version>.json. No such file exists for any version, so the path is skipped either way.
  • _reusable.telemetry-tests.yml:134deep_learning_container.py:281 validates the value as PEP 440. 2.13.0 and 0.29.0 both parse.
  • build-image/compute_ci_tag.sh:32 → the CI tag changes, so these 4 images rebuild once:
    • lambda-vllm-13.0.3-...lambda-vllm-0.29.0-gpu-py313-cu130-<suffix>
    • lambda-pytorch-2.11.0-...lambda-pytorch-2.13.0-gpu-py313-cu130-<suffix>

Build cost

Unlike the metadata-only version of this change, the vLLM source compile does re-run: scripts/ci/build/lambda/lib/source_hash.sh keys the cached wheel on vllm_ref + vllm_version + arch list, and the first two changed. Expect a cold ~85-min compile per vllm image on this PR and on the first build after merge.

Testing

  • generate_release_spec.sh run against the edited configs — container_type unchanged, only version: moves.
  • compute_ci_tag.sh run before/after to confirm the exact tag delta above.
  • pre-commit run --files <the 4 configs> — all applicable hooks pass.
  • uv lock on docker/lambda/pytorch regenerates cleanly with transformers==5.10.4; lock diff is confined to that one package.
  • CI ecr-vulnerability-scan is the real gate for fixes 1–3 and is still running for the pytorch and vllm images at the time of writing.

One knock-on effect worth flagging for whoever looks at dashboards: telemetry for these images splits across the old and new framework_version values at the cutover. That is the intended outcome of the fix.

…mages

pytorch configs claimed 2.11.0 while pyproject pins torch==2.13.0, and the
vllm configs carried the CUDA version (13.0.3) instead of vllm 0.27.1. The
value is baked into bash_telemetry.sh at build time, so usage metrics for
these four images were labelled with the wrong version.

Release tags are unaffected: the lambda entry in DLContainersReleaseLogicPython
frameworks.yml renders release_tag as "{container_type}" and never references
{version}, and its version_pattern is '.*'.

Signed-off-by: Kevin Wang <kwanggg@amazon.com>
@Eren-Jeager123
Eren-Jeager123 marked this pull request as ready for review September 18, 2026 18:55
RUSTSEC-2026-0195 (quick-xml DoS in the bundled uv binary) hit its
2026-09-10 review date, failing every lambda ecr-vulnerability-scan job.
Bump to 2026-12-31, matching the same entry on main (#6738).
Path traversal via save_pretrained() in transformers <=5.8.0.dev0, fixed
in 5.10.0. 5.10.0 itself is yanked upstream, so pin 5.10.4.
RCE in the LlavaOnevision2 processor loader (ignores trust_remote_code),
fixed in 0.28.0; take 0.29.0 as current. flashinfer moves to 0.6.18 to
match 0.29.0's requirements/cuda.txt. The torch trio pin is unchanged
(2.13.0 / 2.11.0 / 0.28.0), so torch-constraints.txt still applies, and
the build-system requires are identical between the two refs.
@Eren-Jeager123 Eren-Jeager123 changed the title fix(lambda): report the real framework version for pytorch and vllm images fix(lambda): report the real framework version, and unblock the ECR scans Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants