From 2392d788b23273125ea7d360e0de1e947e3b1fc1 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 01/59] ci: make one matrix entry one generated package The delivery matrix described build legs indirectly. A row carried a list of suites that resolve-matrix.sh flattened into one leg per suite, and each leg's Debian revision was assembled at resolve time from a per-row debian_version_stub, a matrix-wide suite_suffix_mapping, and a suffix implied by the row's type. Reading the matrix told you which packages existed only after you had run the flattening and the revision formula in your head, and the row's debian_version_suffix field existed purely to write half of that formula down again so a validator could check the two agreed. Split the rows so that one entry is one generated package: a single kernel_variant, type and suite, with its debian_revision stated outright. suite_suffix_mapping and the stub/suffix pair are gone, and with them derive-debian-revision.sh, whose whole job was to apply the formula they encoded. There is nothing left to expand or derive at resolve time, so what an entry says is what gets built. The cost is duplication: nine entries repeat their variant's git_clone, srcpkg, binpkg and kernel_config. They are written out in full rather than sharing YAML anchors, so each entry can be read, grepped and changed on its own, and the resolver takes on the invariants that duplication puts at risk. A variant's entries must agree on srcpkg, binpkg and kernel_config, which decide what the package is; its entries of one type must agree on the kernel ref, so a release ref bump cannot skip a suite and quietly ship one suite a different kernel from its siblings; no two entries may build the same srcpkg at the same revision; and a suite's Daily revision must be its Release revision plus the trailing ~ that sorts it below. Move the matrix to YAML and rewrite the resolver in Python. The resolver was 372 lines whose validation was a single jq program built from elif chains, where each check had to fall through to the next and a new rule meant extending an expression rather than adding a function. The Python collects every problem in one pass and reports them together, so a bad matrix yields its full list of errors instead of the first one jq happened to reach. Both files drop out of the change naturally: the schema is being rewritten anyway, and YAML lets each entry carry the comment that explains it. Unknown fields are now rejected at both entry and root level. Anything left over from this schema change - a stray suites: or suite_suffix_mapping: - fails loudly rather than sitting in the file looking authoritative while nothing reads it. The dkms list moves across with the rest of the schema, validated by resolve-matrix.py the same way kernel_config is and joined into the same comma-separated workflow input. Every entry keeps the kgsl, camx and iris-vpu the row it came from carried, including the ones the Ubuntu override in build-kernel-deb.yml still trims at build time; moving that policy into the matrix is a separate change. Verified against the old resolver across all fifteen filter combinations the workflows can produce: the emitted JSON is identical, so every build leg keeps its inputs and its debian_revision. build-kernel-deb.yml's direct-dispatch path loses the mapping it used to consult, so its debian-version-stub input becomes debian-revision. Left empty it looks up the Daily entry for the variant and suite it was given and builds at the revision the daily build would have used. Dispatching a combination the matrix does not configure now fails instead of inventing a version for it; the error names the override that builds it ad hoc. Signed-off-by: Christopher Obbard --- .github/workflows/build-kernel-deb.yml | 102 ++--- .github/workflows/daily.yml | 45 ++- .github/workflows/pr-build.yml | 10 +- .github/workflows/release.yml | 27 +- .gitignore | 3 + ci/build-matrix.json | 108 ----- ci/build-matrix.yaml | 192 +++++++++ ci/scripts/derive-debian-revision.sh | 131 ------- ci/scripts/resolve-matrix.py | 523 +++++++++++++++++++++++++ ci/scripts/resolve-matrix.sh | 400 ------------------- 10 files changed, 798 insertions(+), 743 deletions(-) delete mode 100644 ci/build-matrix.json create mode 100644 ci/build-matrix.yaml delete mode 100755 ci/scripts/derive-debian-revision.sh create mode 100755 ci/scripts/resolve-matrix.py delete mode 100755 ci/scripts/resolve-matrix.sh diff --git a/.github/workflows/build-kernel-deb.yml b/.github/workflows/build-kernel-deb.yml index d31dd3ba..58e05bef 100644 --- a/.github/workflows/build-kernel-deb.yml +++ b/.github/workflows/build-kernel-deb.yml @@ -50,16 +50,16 @@ on: description: 'Advanced: out-of-tree DKMS modules to build and bundle into the kernel image, comma-separated and without the -dkms suffix (e.g. kgsl,camx); empty bundles nothing' required: false default: '' - debian-version-stub: - description: 'Advanced: Debian version stub; the selected suite''s mapped suffix and a Daily-style trailing ~ are applied automatically' + debian-revision: + description: 'Advanced: Debian revision override (empty takes the Daily revision the matrix gives this variant and suite)' required: false - default: '0qli1' + default: '' localversion: description: 'Advanced: LOCALVERSION override (auto-derived from the resolved ref if empty)' required: false default: '' kver-extra: - description: 'Advanced: extra KVER suffix (e.g. -ci42), appended to the kernel release verbatim' + description: 'Advanced: extra package-version suffix (e.g. -ci42)' required: false default: '' debug-build: @@ -124,13 +124,13 @@ on: debian-revision: description: 'Debian revision component of the package version' type: string - default: '0qli1~' + default: '0qli~' localversion: description: 'Override LOCALVERSION suffix (auto-derived from the resolved ref if empty)' type: string default: '' kver-extra: - description: 'Extra KVER suffix, appended to the kernel release verbatim' + description: 'Extra suffix appended to the package version' type: string default: '' debusine-parent-workspace: @@ -214,7 +214,11 @@ jobs: - name: Checkout pkg-linux-qcom # debian/ and ci/ live on the same branch, so one checkout supplies # both the packaging and resolve-kernel-ref.sh, derive-localversion.sh, - # derive-debian-revision.sh and the suite_suffix_mapping. + # resolve-matrix.py and the delivery matrix it reads. + # + # An empty ref leaves actions/checkout on github.sha, the commit this + # run was dispatched from or the one the calling workflow runs at. + # Credentials are kept: the self-pr step below fetches from origin. uses: actions/checkout@v4 - name: Checkout docker-pkg-build @@ -273,19 +277,7 @@ jobs: echo "KERNEL_SHA=$KERNEL_SHA" >> "$GITHUB_ENV" echo "Kernel HEAD SHA: $KERNEL_SHA" - # Committer date of HEAD, normalised to UTC. Committer rather than - # author date: an author date can be months old on a backported - # patch, which would date the build by when someone first wrote the - # code rather than by when this snapshot came to exist. - KERNEL_DATE=$(TZ=UTC git -C "$KERNEL_DIR" log -1 --format=%cd --date=format-local:%Y%m%d) - [[ "$KERNEL_DATE" =~ ^[0-9]{8}$ ]] || { - echo "::error::could not read a YYYYMMDD commit date from $KERNEL_REF (got '$KERNEL_DATE')" - exit 1 - } - echo "KERNEL_DATE=$KERNEL_DATE" >> "$GITHUB_ENV" - echo "Kernel HEAD commit date (UTC): $KERNEL_DATE" - - - name: Derive version fields + - name: Derive LOCALVERSION env: LOCALVERSION_INPUT: ${{ inputs.localversion }} KERNEL_VARIANT_INPUT: ${{ inputs.kernel-variant || 'qcom-next' }} @@ -293,55 +285,44 @@ jobs: set -euo pipefail OVERRIDE_LV="$LOCALVERSION_INPUT" if [[ -n "$OVERRIDE_LV" ]]; then - # An override supplies no snapshot, and the ref it was chosen to - # replace cannot be trusted to describe it. The Debian version then - # carries no +git; prepare-source.sh warns about the same. echo "Using explicit LOCALVERSION override: $OVERRIDE_LV" echo "LOCALVERSION=$OVERRIDE_LV" >> "$GITHUB_ENV" - echo "SNAPSHOT=" >> "$GITHUB_ENV" - echo "GITSHA=" >> "$GITHUB_ENV" else - # Emits LOCALVERSION=, SNAPSHOT= and GITSHA= lines, already in - # GITHUB_ENV form. - # - # --date is only consulted for branch-tip builds, where the ref - # carries no snapshot of its own. It comes from the commit rather - # than the clock, so rebuilding a commit reproduces its version - # instead of inventing a higher one, and it lands in the same space - # as upstream's tag dates. - FIELDS=$(ci/scripts/derive-localversion.sh \ + LOCALVERSION=$(ci/scripts/derive-localversion.sh \ --variant "$KERNEL_VARIANT_INPUT" \ --ref "$KERNEL_REF" \ - --sha "$KERNEL_SHA" \ - --date "$KERNEL_DATE") - echo "$FIELDS" >> "$GITHUB_ENV" - echo "Derived version fields:" - echo "$FIELDS" + --sha "$(echo "$KERNEL_SHA" | cut -c1-12)") + echo "LOCALVERSION=$LOCALVERSION" >> "$GITHUB_ENV" + echo "Derived LOCALVERSION: $LOCALVERSION" fi - name: Resolve Debian revision - # workflow_call callers (daily.yml, release.yml) pass an already - # fully-derived debian-revision from resolve-matrix.sh. Direct - # workflow_dispatch runs pass only debian-version-stub and get the - # selected suite's mapped suffix applied automatically, always using - # Daily semantics since direct dispatch is build-only and - # non-promoting. + # workflow_call callers (daily.yml, release.yml) pass the entry's own + # debian-revision straight from the matrix. A direct workflow_dispatch + # that leaves it empty looks the same entry up itself, so a manual + # build of a configured variant and suite gets exactly the revision the + # daily build would have used. The lookup is always the Daily entry: + # direct dispatch is build-only and never promotes. env: DEBIAN_REVISION_INPUT: ${{ inputs.debian-revision }} - DEBIAN_VERSION_STUB_INPUT: ${{ inputs.debian-version-stub || '0qli1' }} + KERNEL_VARIANT_INPUT: ${{ inputs.kernel-variant || 'qcom-next' }} run: | set -euo pipefail INPUT_REVISION="$DEBIAN_REVISION_INPUT" if [[ -n "$INPUT_REVISION" ]]; then DEBIAN_REVISION="$INPUT_REVISION" - echo "Using matrix-derived Debian revision: $DEBIAN_REVISION" + echo "Using supplied Debian revision: $DEBIAN_REVISION" else - DEBIAN_REVISION=$(ci/scripts/derive-debian-revision.sh \ - --stub "$DEBIAN_VERSION_STUB_INPUT" \ + DEBIAN_REVISION=$(ci/scripts/resolve-matrix.py \ + --type Daily \ + --kernel-variant "$KERNEL_VARIANT_INPUT" \ --suite "$DISTRO" \ - --delivery-type Daily \ - --matrix-file ci/build-matrix.json) - echo "Derived Debian revision for direct dispatch: $DEBIAN_REVISION" + --field debian_revision) || { + echo "::error::No Daily matrix entry for kernel-variant=$KERNEL_VARIANT_INPUT suite=$DISTRO." + echo "::error::Add one to ci/build-matrix.yaml, or set the debian-revision input to build this combination ad hoc." + exit 1 + } + echo "Matrix Debian revision for $KERNEL_VARIANT_INPUT/$DISTRO: $DEBIAN_REVISION" fi echo "DEBIAN_REVISION=$DEBIAN_REVISION" >> "$GITHUB_ENV" @@ -411,10 +392,10 @@ jobs: # A listed module is a presence contract, so an Ubuntu leg that asked # for them would hard-fail rather than quietly ship without them. # - # This is delivery policy and belongs in ci/build-matrix.json, which - # cannot express a per-suite dkms list yet. Revert this override — - # and the "needs: resolve"/SUITE_FAMILY plumbing that feeds it — once - # the matrix can. + # This is delivery policy and belongs in ci/build-matrix.yaml, whose + # entries are now one per suite and can carry it. Revert this + # override — and the "needs: resolve"/SUITE_FAMILY plumbing that + # feeds it — in the commit that moves the policy there. if [[ "$SUITE_FAMILY" == "ubuntu" ]]; then DKMS_LIST="kgsl" echo "Ubuntu build: DKMS list overridden to '$DKMS_LIST' (was '$DKMS_INPUT')" @@ -428,16 +409,7 @@ jobs: --debian-revision "$DEBIAN_REVISION" ) - # All three version fields come from derive-localversion.sh, which - # composed them from the ref and HEAD together. Nothing here appends - # to them: the SHA is already inside LOCALVERSION, and GITSHA carries - # it separately for the Debian version, which joins it differently. [[ -n "$LOCALVERSION" ]] && ARGS+=(--localversion "$LOCALVERSION") - [[ -n "$SNAPSHOT" ]] && ARGS+=(--snapshot "$SNAPSHOT") - # Full SHA: prepare-source.sh takes the first 12 for the version - # strings and records the whole thing in the changelog, alongside the - # repository and ref that the version strings do not name. - ARGS+=(--git-sha "$KERNEL_SHA" --git-clone "$KERNEL_URL" --git-ref "$KERNEL_REF") [[ -n "$KVER_EXTRA_INPUT" ]] && ARGS+=(--kver-extra "$KVER_EXTRA_INPUT") [[ -n "$KERNEL_CONFIG_INPUT" ]] && ARGS+=(--kernel-config "$KERNEL_CONFIG_INPUT") # Always passed, unlike the optional inputs above: an empty list is a diff --git a/.github/workflows/daily.yml b/.github/workflows/daily.yml index a3022f7e..1e30522c 100644 --- a/.github/workflows/daily.yml +++ b/.github/workflows/daily.yml @@ -1,20 +1,20 @@ name: daily # Daily kernel package builds. Runs at 3:00 PM PST (23:00 UTC), one parallel -# build per Daily entry in ci/build-matrix.json after suite flattening. -# Manual dispatch can run the full Daily matrix, every suite for one kernel -# variant, or one kernel variant and suite. Variant and suite are matrix values, -# so future variants need no workflow edit. +# build per Daily entry in ci/build-matrix.yaml. Manual dispatch can run the +# full Daily matrix, every suite for one kernel variant, or one kernel variant +# and suite. Variant and suite are matrix values, so future variants need no +# workflow edit. # -# Matrix schema: ci/build-matrix.json -# Each row with "type": "Daily" is expanded by ci/scripts/resolve-matrix.sh: -# the "suites" array is flattened into one job per suite. All other fields -# (kernel_variant, srcpkg, binpkg, kernel_config, dkms, debian_revision, -# git_clone, branch_or_tag, ref_strategy) are passed through to -# build-kernel-deb.yml as inputs. +# Matrix schema: ci/build-matrix.yaml +# One entry is one generated package, so ci/scripts/resolve-matrix.py selects +# the entries with type: Daily and passes them through as they stand. Every +# field (kernel_variant, suite, srcpkg, binpkg, kernel_config, dkms, +# debian_revision, git_clone, branch_or_tag, ref_strategy) becomes a +# build-kernel-deb.yml input. # -# To add a daily build target: add a row to ci/build-matrix.json with -# "type": "Daily". No workflow changes needed. +# To add a daily build target: add an entry to ci/build-matrix.yaml with +# type: Daily. No workflow changes needed. on: schedule: @@ -48,12 +48,11 @@ permissions: jobs: # --------------------------------------------------------------------------- - # configure-matrix: flatten the Daily rows from ci/build-matrix.json. + # configure-matrix: select the Daily entries from ci/build-matrix.yaml. # - # Delegates to ci/scripts/resolve-matrix.sh which filters type=Daily rows - # and expands each row's "suites" array into one flat entry per suite. - # Manual dispatch selects the complete matrix, every suite for a variant, or - # one variant and suite. + # Delegates to ci/scripts/resolve-matrix.py, which validates the whole matrix + # and returns the entries with type: Daily. Manual dispatch selects the + # complete matrix, every suite for a variant, or one variant and suite. # --------------------------------------------------------------------------- configure-matrix: name: Configure build matrix @@ -80,29 +79,29 @@ jobs: case "$BUILD_SCOPE" in "Full matrix") - echo "Source: ci/build-matrix.json (full Daily matrix)" + echo "Source: ci/build-matrix.yaml (full Daily matrix)" args=(--type Daily) ;; "Selected variant (all suites)") - echo "Source: ci/build-matrix.json (variant: $INPUT_VARIANT, all suites)" + echo "Source: ci/build-matrix.yaml (variant: $INPUT_VARIANT, all suites)" args=(--type Daily --kernel-variant "$INPUT_VARIANT") ;; "Selected variant and suite") - echo "Source: ci/build-matrix.json (variant: $INPUT_VARIANT, suite: $INPUT_SUITE)" - args=(--type Daily --kernel-variant "$INPUT_VARIANT" --single-suite "$INPUT_SUITE") + echo "Source: ci/build-matrix.yaml (variant: $INPUT_VARIANT, suite: $INPUT_SUITE)" + args=(--type Daily --kernel-variant "$INPUT_VARIANT" --suite "$INPUT_SUITE") ;; *) echo "ERROR: unsupported build scope: $BUILD_SCOPE" >&2 exit 1 ;; esac - MATRIX=$(ci/scripts/resolve-matrix.sh "${args[@]}") + MATRIX=$(ci/scripts/resolve-matrix.py "${args[@]}") echo "matrix=$MATRIX" >> "$GITHUB_OUTPUT" echo "Matrix entries: $(echo "$MATRIX" | jq length)" # --------------------------------------------------------------------------- - # build: one parallel job per flat matrix entry. + # build: one parallel job per matrix entry. # # Each entry calls build-kernel-deb.yml as a reusable workflow. The full # set of matrix fields (suite, srcpkg, binpkg, kernel_config, dkms, etc.) is diff --git a/.github/workflows/pr-build.yml b/.github/workflows/pr-build.yml index 4f9e5e3a..2d2888ae 100644 --- a/.github/workflows/pr-build.yml +++ b/.github/workflows/pr-build.yml @@ -1,6 +1,10 @@ name: PR Build # Build-test PRs through the same pipeline the daily builds use. +# +# Packaging and CI now live on one branch, so a PR is tested entirely against +# its own commit: its ci/build-matrix.yaml defines the legs, and its debian/ +# tree and ci/scripts build them. Nothing is read from main. on: pull_request: @@ -16,7 +20,7 @@ permissions: jobs: # --------------------------------------------------------------------------- - # configure-matrix: flatten the Daily rows from the PR's build-matrix.json. + # configure-matrix: select the Daily entries from the PR's build-matrix.yaml. # # The PR head, not the merge commit: a push mid-run must not change what this # run tested. A SHA also resolves for fork PRs, whose head branch does not @@ -37,12 +41,12 @@ jobs: id: set-matrix run: | set -euo pipefail - MATRIX=$(ci/scripts/resolve-matrix.sh --type Daily) + MATRIX=$(ci/scripts/resolve-matrix.py --type Daily) echo "matrix=$MATRIX" >> "$GITHUB_OUTPUT" echo "Matrix entries: $(echo "$MATRIX" | jq length)" # --------------------------------------------------------------------------- - # build: one parallel job per flat matrix entry, mirroring daily.yml. + # build: one parallel job per matrix entry, mirroring daily.yml. # # fail-fast is disabled so a broken trixie leg still leaves forky and # resolute results on the PR. diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 32eddcf5..62e31eea 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -2,14 +2,15 @@ name: release # Release kernel package builds. Triggered manually only. # -# Reads Release rows from ci/build-matrix.json, flattens suites, and builds -# each entry using the matrix-selected ref strategy and branch_or_tag. After a -# successful build, packages are promoted from the ephemeral Debusine CI -# workspace to the stable target workspace (e.g. qli) via debusine-action -# lib/release. +# Reads the Release entries from ci/build-matrix.yaml and builds each one using +# the matrix-selected ref strategy and branch_or_tag. After a successful build, +# packages are promoted from the ephemeral Debusine CI workspace to the stable +# target workspace (e.g. qli) via debusine-action lib/release. # -# To update a release ref: open a PR that changes the Release row for that -# kernel variant in ci/build-matrix.json. +# To update a release ref: open a PR that changes branch_or_tag on every +# Release entry for that kernel variant in ci/build-matrix.yaml. A variant's +# Release entries must agree on the ref, so resolve-matrix.py rejects the +# matrix if only some of them are updated. # # This workflow intentionally has no schedule trigger. Releases are always # initiated manually after validation. @@ -42,7 +43,7 @@ permissions: jobs: # --------------------------------------------------------------------------- - # configure-matrix: flatten the Release rows from ci/build-matrix.json. + # configure-matrix: select the Release entries from ci/build-matrix.yaml. # --------------------------------------------------------------------------- configure-matrix: name: Configure release matrix @@ -67,25 +68,25 @@ jobs: case "$RELEASE_SCOPE" in "Selected variant (all suites)") - echo "Source: ci/build-matrix.json (variant: $INPUT_VARIANT, all suites)" + echo "Source: ci/build-matrix.yaml (variant: $INPUT_VARIANT, all suites)" args=(--type Release --kernel-variant "$INPUT_VARIANT") ;; "Selected variant and suite") - echo "Source: ci/build-matrix.json (variant: $INPUT_VARIANT, suite: $INPUT_SUITE)" - args=(--type Release --kernel-variant "$INPUT_VARIANT" --single-suite "$INPUT_SUITE") + echo "Source: ci/build-matrix.yaml (variant: $INPUT_VARIANT, suite: $INPUT_SUITE)" + args=(--type Release --kernel-variant "$INPUT_VARIANT" --suite "$INPUT_SUITE") ;; *) echo "ERROR: unsupported release scope: $RELEASE_SCOPE" >&2 exit 1 ;; esac - MATRIX=$(ci/scripts/resolve-matrix.sh "${args[@]}") + MATRIX=$(ci/scripts/resolve-matrix.py "${args[@]}") echo "matrix=$MATRIX" >> "$GITHUB_OUTPUT" echo "Matrix entries: $(echo "$MATRIX" | jq length)" # --------------------------------------------------------------------------- - # build: one parallel job per flat Release matrix entry. + # build: one parallel job per Release matrix entry. # # Uses the ref strategy and branch_or_tag selected by the matrix. # Passes target_workspace so build-kernel-debusine.yml triggers the Debusine diff --git a/.gitignore b/.gitignore index 2583b54a..d2fc2efd 100644 --- a/.gitignore +++ b/.gitignore @@ -18,3 +18,6 @@ debian/linux-image-*.prerm # Default kernel source and build output directories kernel-source/ kernel-build/ + +# Python bytecode cache from ci/scripts/ +__pycache__/ diff --git a/ci/build-matrix.json b/ci/build-matrix.json deleted file mode 100644 index a28175d4..00000000 --- a/ci/build-matrix.json +++ /dev/null @@ -1,108 +0,0 @@ -{ - "suite_suffix_mapping": { - "trixie": "~bpo13+1", - "forky": "", - "resolute": "~26.04.1" - }, - "deliveries": [ - { - "kernel_variant": "qcom-next", - "type": "Daily", - "suites": [ - "trixie", - "forky", - "resolute" - ], - "git_clone": "https://github.com/qualcomm-linux/kernel", - "branch_or_tag": "qcom-next", - "ref_strategy": "latest_tag", - "tag_pattern": "qcom-next-*", - "srcpkg": "linux-qcom-next", - "binpkg": "linux-image-qcom-next", - "kernel_config": [], - "dkms": [ - "kgsl", - "camx", - "iris-vpu", - "audioreach" - ], - "debian_version_stub": "0qli1", - "debian_version_suffix": "~" - }, - { - "kernel_variant": "qcom-next", - "type": "Release", - "target_workspace": "qli", - "suites": [ - "trixie", - "forky" - ], - "git_clone": "https://github.com/qualcomm-linux/kernel", - "branch_or_tag": "qcom-next-7.2-20260826", - "ref_strategy": "pinned_ref", - "srcpkg": "linux-qcom-next", - "binpkg": "linux-image-qcom-next", - "kernel_config": [], - "dkms": [ - "kgsl", - "camx", - "iris-vpu", - "audioreach" - ], - "debian_version_stub": "0qli1", - "debian_version_suffix": "" - }, - { - "kernel_variant": "qcom-next-debug", - "type": "Daily", - "suites": [ - "trixie", - "forky" - ], - "git_clone": "https://github.com/qualcomm-linux/kernel", - "branch_or_tag": "qcom-next", - "ref_strategy": "latest_tag", - "tag_pattern": "qcom-next-*", - "srcpkg": "linux-qcom-next-debug", - "binpkg": "linux-image-qcom-next-debug", - "kernel_config": [ - "intree:arch/arm64/configs/qcom_debug.config", - "intree:kernel/configs/debug.config" - ], - "dkms": [ - "kgsl", - "camx", - "iris-vpu", - "audioreach" - ], - "debian_version_stub": "0qli1", - "debian_version_suffix": "~" - }, - { - "kernel_variant": "qcom-next-debug", - "type": "Release", - "target_workspace": "qli", - "suites": [ - "trixie", - "forky" - ], - "git_clone": "https://github.com/qualcomm-linux/kernel", - "branch_or_tag": "qcom-next-7.2-20260826", - "ref_strategy": "pinned_ref", - "srcpkg": "linux-qcom-next-debug", - "binpkg": "linux-image-qcom-next-debug", - "kernel_config": [ - "intree:arch/arm64/configs/qcom_debug.config", - "intree:kernel/configs/debug.config" - ], - "dkms": [ - "kgsl", - "camx", - "iris-vpu", - "audioreach" - ], - "debian_version_stub": "0qli1", - "debian_version_suffix": "" - } - ] -} diff --git a/ci/build-matrix.yaml b/ci/build-matrix.yaml new file mode 100644 index 00000000..208beb61 --- /dev/null +++ b/ci/build-matrix.yaml @@ -0,0 +1,192 @@ +# Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. +# SPDX-License-Identifier: BSD-3-Clause-Clear +# +# Kernel delivery matrix. +# +# One entry in "deliveries" is one generated package: a single kernel_variant, +# a single delivery type, and a single suite. The only list-valued fields are +# kernel_config and dkms, which name the fragments and the out-of-tree modules +# that go into that one package, so an entry needs no expansion — ci/scripts/ +# resolve-matrix.py validates the document, filters it, and hands the matching +# entries straight to the workflow matrix. +# +# Entries are written out in full rather than sharing YAML anchors, so each one +# can be read, grepped, and changed on its own. resolve-matrix.py enforces the +# consistency that duplication would otherwise put at risk: a variant's entries +# must agree on srcpkg, binpkg and kernel_config, and its entries of one +# delivery type must agree on the kernel ref they build. +# +# debian_revision is stated outright, not derived. Daily revisions end in "~" +# so they sort below the Release revision for the same suite; the rest of the +# revision places the suite where it belongs relative to the other suites (see +# the ordering discussion in README.md). No two entries may produce the same +# srcpkg at the same debian_revision. + +deliveries: + # --------------------------------------------------------------------------- + # qcom-next — the standard kernel. + # --------------------------------------------------------------------------- + - kernel_variant: qcom-next + type: Daily + suite: trixie + git_clone: https://github.com/qualcomm-linux/kernel + branch_or_tag: qcom-next + ref_strategy: latest_tag + tag_pattern: 'qcom-next-*' + srcpkg: linux-qcom-next + binpkg: linux-image-qcom-next + kernel_config: [] + dkms: + - kgsl + - camx + - iris-vpu + - audioreach + debian_revision: '0qli1~bpo13+1~' + + - kernel_variant: qcom-next + type: Daily + suite: forky + git_clone: https://github.com/qualcomm-linux/kernel + branch_or_tag: qcom-next + ref_strategy: latest_tag + tag_pattern: 'qcom-next-*' + srcpkg: linux-qcom-next + binpkg: linux-image-qcom-next + kernel_config: [] + dkms: + - kgsl + - camx + - iris-vpu + - audioreach + debian_revision: '0qli1~' + + - kernel_variant: qcom-next + type: Daily + suite: resolute + git_clone: https://github.com/qualcomm-linux/kernel + branch_or_tag: qcom-next + ref_strategy: latest_tag + tag_pattern: 'qcom-next-*' + srcpkg: linux-qcom-next + binpkg: linux-image-qcom-next + kernel_config: [] + dkms: + - kgsl + - camx + - iris-vpu + - audioreach + debian_revision: '0qli1~26.04.1~' + + - kernel_variant: qcom-next + type: Release + suite: trixie + git_clone: https://github.com/qualcomm-linux/kernel + branch_or_tag: qcom-next-7.2-20260826 + ref_strategy: pinned_ref + srcpkg: linux-qcom-next + binpkg: linux-image-qcom-next + kernel_config: [] + dkms: + - kgsl + - camx + - iris-vpu + - audioreach + debian_revision: '0qli1~bpo13+1' + target_workspace: qli + + - kernel_variant: qcom-next + type: Release + suite: forky + git_clone: https://github.com/qualcomm-linux/kernel + branch_or_tag: qcom-next-7.2-20260826 + ref_strategy: pinned_ref + srcpkg: linux-qcom-next + binpkg: linux-image-qcom-next + kernel_config: [] + dkms: + - kgsl + - camx + - iris-vpu + - audioreach + debian_revision: '0qli1' + target_workspace: qli + + # --------------------------------------------------------------------------- + # qcom-next-debug — the same kernel ref built with the in-tree debug + # fragments, packaged separately so it installs alongside qcom-next. + # --------------------------------------------------------------------------- + - kernel_variant: qcom-next-debug + type: Daily + suite: trixie + git_clone: https://github.com/qualcomm-linux/kernel + branch_or_tag: qcom-next + ref_strategy: latest_tag + tag_pattern: 'qcom-next-*' + srcpkg: linux-qcom-next-debug + binpkg: linux-image-qcom-next-debug + kernel_config: + - intree:arch/arm64/configs/qcom_debug.config + - intree:kernel/configs/debug.config + dkms: + - kgsl + - camx + - iris-vpu + - audioreach + debian_revision: '0qli1~bpo13+1~' + + - kernel_variant: qcom-next-debug + type: Daily + suite: forky + git_clone: https://github.com/qualcomm-linux/kernel + branch_or_tag: qcom-next + ref_strategy: latest_tag + tag_pattern: 'qcom-next-*' + srcpkg: linux-qcom-next-debug + binpkg: linux-image-qcom-next-debug + kernel_config: + - intree:arch/arm64/configs/qcom_debug.config + - intree:kernel/configs/debug.config + dkms: + - kgsl + - camx + - iris-vpu + - audioreach + debian_revision: '0qli1~' + + - kernel_variant: qcom-next-debug + type: Release + suite: trixie + git_clone: https://github.com/qualcomm-linux/kernel + branch_or_tag: qcom-next-7.2-20260826 + ref_strategy: pinned_ref + srcpkg: linux-qcom-next-debug + binpkg: linux-image-qcom-next-debug + kernel_config: + - intree:arch/arm64/configs/qcom_debug.config + - intree:kernel/configs/debug.config + dkms: + - kgsl + - camx + - iris-vpu + - audioreach + debian_revision: '0qli1~bpo13+1' + target_workspace: qli + + - kernel_variant: qcom-next-debug + type: Release + suite: forky + git_clone: https://github.com/qualcomm-linux/kernel + branch_or_tag: qcom-next-7.2-20260826 + ref_strategy: pinned_ref + srcpkg: linux-qcom-next-debug + binpkg: linux-image-qcom-next-debug + kernel_config: + - intree:arch/arm64/configs/qcom_debug.config + - intree:kernel/configs/debug.config + dkms: + - kgsl + - camx + - iris-vpu + - audioreach + debian_revision: '0qli1' + target_workspace: qli diff --git a/ci/scripts/derive-debian-revision.sh b/ci/scripts/derive-debian-revision.sh deleted file mode 100755 index 008e993e..00000000 --- a/ci/scripts/derive-debian-revision.sh +++ /dev/null @@ -1,131 +0,0 @@ -#!/bin/bash -# Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. -# SPDX-License-Identifier: BSD-3-Clause-Clear -set -euo pipefail - -# Derive the suite-specific Debian revision for one delivery leg. -# -# Formula: -# debian_revision = stub + suite_suffix_mapping[suite] + delivery_suffix -# delivery_suffix: Daily -> "~", Release -> "" -# -# This is the single implementation of the formula. It is called both by -# resolve-matrix.sh (once per flattened Daily/Release leg) and by -# build-kernel-deb.yml's direct-dispatch path (one suite, no full matrix -# context), so the derivation and its validation live in exactly one place. -# -# Usage: -# ci/scripts/derive-debian-revision.sh --stub 0qli1 --suite trixie --delivery-type Daily -# ci/scripts/derive-debian-revision.sh --stub 0qli1 --suite forky --delivery-type Release --matrix-file ci/build-matrix.json -# -# Options: -# --stub STUB Debian version stub, e.g. 0qli1. Must end in a digit: -# that digit is the packaging revision, bumped for a -# rebuild of an unchanged kernel snapshot. Must not -# end in ~ (the delivery suffix supplies any -# trailing ~). Required. -# --suite SUITE Target suite; must have an entry in -# suite_suffix_mapping. Required. -# --delivery-type TYPE Daily or Release. Required. -# --matrix-file FILE Path to the matrix JSON containing -# suite_suffix_mapping -# (default: ci/build-matrix.json relative to CWD). -# -# Output: -# Final Debian revision printed to stdout. -# -# Exit codes: -# 0 Success. -# 1 Error (invalid args, malformed or missing suite_suffix_mapping, -# unmapped suite, unsupported delivery type). - -STUB="" -SUITE="" -DELIVERY_TYPE="" -MATRIX_FILE="ci/build-matrix.json" - -usage() { - sed -n '/^# Usage:/,/^$/p' "$0" | sed 's/^# \?//' - exit 1 -} - -while [[ $# -gt 0 ]]; do - case $1 in - --stub) STUB="$2"; shift 2 ;; - --suite) SUITE="$2"; shift 2 ;; - --delivery-type) DELIVERY_TYPE="$2"; shift 2 ;; - --matrix-file) MATRIX_FILE="$2"; shift 2 ;; - -h|--help) usage ;; - *) echo "ERROR: Unknown option: $1" >&2; usage ;; - esac -done - -[[ -n "$STUB" ]] || { echo "ERROR: --stub is required" >&2; exit 1; } -[[ -n "$SUITE" ]] || { echo "ERROR: --suite is required" >&2; exit 1; } -[[ -n "$DELIVERY_TYPE" ]] || { echo "ERROR: --delivery-type is required" >&2; exit 1; } -[[ "$STUB" != *"~" ]] || { echo "ERROR: --stub must not end in ~ (got '$STUB')" >&2; exit 1; } -# The trailing digit is the packaging revision: it is the only field left to -# bump when the kernel snapshot is unchanged but the packaging is rebuilt. -# suite_suffix_mapping is a per-suite constant and cannot carry it, and the -# delivery suffix is the Daily/Release marker, so a stub without a digit leaves -# a rebuild with nowhere to go. -[[ "$STUB" =~ [0-9]$ ]] || { - echo "ERROR: --stub must end in a digit, the packaging revision (got '$STUB'; use '${STUB}1')" >&2 - exit 1 -} -[[ -f "$MATRIX_FILE" ]] || { echo "ERROR: Matrix file not found: $MATRIX_FILE" >&2; exit 1; } - -jq empty "$MATRIX_FILE" 2>/dev/null \ - || { echo "ERROR: Invalid JSON in $MATRIX_FILE" >&2; exit 1; } - -mapping_errors=$(jq -r ' - .suite_suffix_mapping as $mapping | - if ($mapping | type) != "object" - then "suite_suffix_mapping is missing or not an object" - else - ( - [$mapping | to_entries[] | select(.value | type != "string") | "suite_suffix_mapping[" + .key + "] must be a string"] - + - [ - $mapping - | to_entries[] - | select((.value | type == "string") and .value != "" and (.value | test("^~") | not)) - | "suite_suffix_mapping[" + .key + "] must be empty or start with ~ (got \"" + .value + "\")" - ] - + - [ - $mapping - | to_entries - | group_by(.value) - | map(select(length > 1)) - | .[]? - | "suites " + ([.[].key] | join(", ")) + " share the same suffix \"" + .[0].value + "\"" - ] - ) | .[] - end -' "$MATRIX_FILE") - -if [[ -n "$mapping_errors" ]]; then - echo "ERROR: Invalid suite_suffix_mapping in $MATRIX_FILE:" >&2 - while IFS= read -r error; do - [[ -n "$error" ]] && echo " - $error" >&2 - done <<< "$mapping_errors" - exit 1 -fi - -SUFFIX=$(jq -r --arg suite "$SUITE" '.suite_suffix_mapping[$suite] // "__MISSING__"' "$MATRIX_FILE") -[[ "$SUFFIX" != "__MISSING__" ]] || { - echo "ERROR: no suite_suffix_mapping entry for suite '$SUITE'" >&2 - exit 1 -} - -case "$DELIVERY_TYPE" in - Daily) DELIVERY_SUFFIX="~" ;; - Release) DELIVERY_SUFFIX="" ;; - *) - echo "ERROR: --delivery-type must be Daily or Release (got '$DELIVERY_TYPE')" >&2 - exit 1 - ;; -esac - -echo "${STUB}${SUFFIX}${DELIVERY_SUFFIX}" diff --git a/ci/scripts/resolve-matrix.py b/ci/scripts/resolve-matrix.py new file mode 100755 index 00000000..4f97703c --- /dev/null +++ b/ci/scripts/resolve-matrix.py @@ -0,0 +1,523 @@ +#!/usr/bin/env python3 +# Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. +# SPDX-License-Identifier: BSD-3-Clause-Clear +"""Validate and select entries from the kernel delivery matrix. + +ci/build-matrix.yaml holds one entry per generated package: a single +kernel_variant, a single delivery type, and a single suite. Nothing here +expands or derives anything -- the matrix is already flat, and each entry +states its own debian_revision. This script validates the whole document, +selects the entries a caller asked for, and prints them. + +The document is validated in full on every invocation, not just the selected +entries, so a typo in a Release entry fails the daily build too rather than +lying in wait until someone runs a release. + +Usage: + ci/scripts/resolve-matrix.py --type Daily + ci/scripts/resolve-matrix.py --type Release --kernel-variant qcom-next + ci/scripts/resolve-matrix.py --type Daily --kernel-variant qcom-next --suite trixie + ci/scripts/resolve-matrix.py --type Daily --kernel-variant qcom-next \\ + --suite trixie --field debian_revision + +Options: + --type TYPE Delivery type to select (Daily or Release). + Required. + --kernel-variant VARIANT Select only this kernel variant. + --suite SUITE Select only this suite. + --field NAME Print just this field of the single selected + entry, unquoted. Errors unless exactly one + entry matches. + --matrix-file FILE Matrix path (default: ci/build-matrix.yaml + relative to CWD). + +Output: + Without --field, a compact JSON array of the selected entries, ready for a + GitHub Actions matrix `include`. kernel_config and dkms are joined into the + comma-separated strings that build-kernel-deb.yml's kernel-config and dkms + inputs, and prepare-source.sh's --kernel-config and --dkms, expect; every + other field is passed through as written. + + With --field, the named field's value alone, so a workflow step can capture + it directly. + +Exit codes: + 0 Success, at least one entry selected. + 1 Error (invalid arguments, matrix validation failure, no matching entry, + or --field matching more than one entry). +""" + +import argparse +import json +import re +import sys +from collections import defaultdict + +try: + import yaml +except ImportError: + sys.exit( + "ERROR: PyYAML is required to read the delivery matrix.\n" + " Install it with 'apt-get install python3-yaml' or 'pip install pyyaml'." + ) + +DEFAULT_MATRIX_FILE = "ci/build-matrix.yaml" + +DELIVERY_TYPES = ("Daily", "Release") +REF_STRATEGIES = ("latest_tag", "branch_tip", "pinned_ref") + +# A delivery type constrains how its kernel ref is chosen: a Daily build tracks +# something moving, a Release build is pinned to an immutable ref. +REF_STRATEGIES_FOR_TYPE = { + "Daily": ("latest_tag", "branch_tip"), + "Release": ("pinned_ref",), +} + +REQUIRED_STRING_FIELDS = ( + "kernel_variant", + "type", + "suite", + "git_clone", + "branch_or_tag", + "ref_strategy", + "srcpkg", + "binpkg", + "debian_revision", +) + +OPTIONAL_STRING_FIELDS = ( + "tag_pattern", + "target_workspace", + "localversion", + "kver_extra", + "debusine_parent_workspace", +) + +KNOWN_FIELDS = frozenset( + REQUIRED_STRING_FIELDS + OPTIONAL_STRING_FIELDS + ("kernel_config", "dkms") +) + +# Fields that identify the variant itself rather than one of its build legs. +# Every entry for a variant must agree on them, because they decide what the +# package is; the entries only differ in where it is delivered. +VARIANT_IDENTITY_FIELDS = ("srcpkg", "binpkg", "kernel_config", "dkms") + +# Fields deciding which kernel tree is built. All entries for one variant and +# delivery type build the same source, so a stale suite cannot quietly ship a +# different kernel from its siblings. +REF_FIELDS = ("git_clone", "branch_or_tag", "ref_strategy", "tag_pattern") + +NAME_RE = re.compile(r"^[a-z0-9]+(?:[a-z0-9-]*[a-z0-9])?$") + +# A Debian revision: no hyphen (that would start a new revision component) and +# none of the characters dpkg rejects in a version. +REVISION_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9.+~]*$") + +# An "intree:" entry names a fragment shipped by the kernel source, as a path +# relative to the kernel source root. A bare entry names a fragment in +# debian/config-available/, with or without its .config extension. +INTREE_PATH_RE = re.compile(r"^([A-Za-z0-9_.-]+/)*[A-Za-z0-9_.-]+\.config$") +BARE_FRAGMENT_RE = re.compile(r"^[A-Za-z0-9_.-]+$") + +# A dkms entry is a package name stem: the build wants -dkms available, +# and generates both the Build-Depends entry and the debian/dkms-modules +# manifest from it. Same shape debian/rules enforces at prepare time, checked +# here so a typo fails before a runner is claimed rather than mid-build. +DKMS_NAME_RE = re.compile(r"^[a-z0-9][a-z0-9+.-]*$") + + +def fragment_basename(fragment): + """Filename a fragment lands under in debian/config/, minus .config. + + Every fragment is copied into debian/config/ under its basename, so + arch/arm64/configs/hardening.config and kernel/configs/hardening.config + collide there even though the entries differ. + """ + return fragment.removeprefix("intree:").rsplit("/", 1)[-1].removesuffix(".config") + + +def check_kernel_config(entry, report): + """Validate one entry's kernel_config list.""" + fragments = entry.get("kernel_config") + if not isinstance(fragments, list): + report("kernel_config must be a list") + return + if any(not isinstance(f, str) or not f for f in fragments): + report("kernel_config must contain only non-empty strings") + return + if any("," in f for f in fragments): + report( + "kernel_config entries must not contain commas; " + "use one list element per fragment" + ) + return + if len(set(fragments)) != len(fragments): + report("kernel_config must not contain duplicates") + + for fragment in fragments: + if fragment.startswith("intree:"): + path = fragment.removeprefix("intree:") + traversal = path == ".." or path.startswith("../") or "/../" in path + if not INTREE_PATH_RE.match(path) or traversal or path.endswith("/.."): + report( + f"intree: entry '{fragment}' must be a kernel-source-relative " + "path ending in .config " + "(e.g. intree:arch/arm64/configs/qcom_debug.config)" + ) + elif not BARE_FRAGMENT_RE.match(fragment): + report( + f"kernel_config entry '{fragment}' must be a fragment name from " + "debian/config-available/ or an intree: path" + ) + + basenames = [fragment_basename(f) for f in fragments] + if len(set(basenames)) != len(basenames): + report("kernel_config entries must not resolve to the same fragment filename") + + +def check_dkms(entry, report): + """Validate one entry's dkms list. + + An empty list is meaningful: it bundles nothing, as opposed to leaving the + field out, which the matrix does not allow. + """ + modules = entry.get("dkms") + if not isinstance(modules, list): + report("dkms must be a list") + return + if any(not isinstance(m, str) or not m for m in modules): + report("dkms must contain only non-empty strings") + return + if len(set(modules)) != len(modules): + report("dkms must not contain duplicates") + + for module in modules: + if module.endswith("-dkms"): + report( + f"dkms entry '{module}' must omit the -dkms suffix " + f"(use '{module.removesuffix('-dkms')}')" + ) + elif not DKMS_NAME_RE.match(module): + report(f"dkms entry '{module}' must be a package name stem, e.g. kgsl") + + +def check_entry(entry, report): + """Validate one delivery entry in isolation.""" + for field in REQUIRED_STRING_FIELDS: + value = entry.get(field) + if not isinstance(value, str) or not value: + report(f"missing or invalid {field}") + + for field in OPTIONAL_STRING_FIELDS: + if field in entry and not isinstance(entry[field], str): + report(f"invalid {field}") + + for field in sorted(set(entry) - KNOWN_FIELDS): + report(f"unknown field {field}") + + for field in ("kernel_variant", "suite"): + value = entry.get(field) + if isinstance(value, str) and not NAME_RE.match(value): + report(f"{field} must use lowercase letters, digits, and internal hyphens") + + check_kernel_config(entry, report) + check_dkms(entry, report) + + delivery_type = entry.get("type") + if delivery_type not in DELIVERY_TYPES: + report("type must be Daily or Release") + + ref_strategy = entry.get("ref_strategy") + if ref_strategy not in REF_STRATEGIES: + report("ref_strategy must be " + ", ".join(REF_STRATEGIES)) + elif delivery_type in REF_STRATEGIES_FOR_TYPE: + allowed = REF_STRATEGIES_FOR_TYPE[delivery_type] + if ref_strategy not in allowed: + report( + f"{delivery_type} entries must use " + + " or ".join(f"ref_strategy={s}" for s in allowed) + ) + + if ref_strategy == "latest_tag": + if not entry.get("tag_pattern"): + report("missing or invalid tag_pattern") + elif "tag_pattern" in entry: + report("tag_pattern is only valid with ref_strategy=latest_tag") + + if delivery_type == "Release": + if not entry.get("target_workspace"): + report("missing or invalid target_workspace") + elif "target_workspace" in entry: + report("target_workspace is only valid for Release") + + revision = entry.get("debian_revision") + if isinstance(revision, str) and revision: + if not REVISION_RE.match(revision): + report( + f'debian_revision "{revision}" is not a valid Debian revision ' + "(letters, digits, and . + ~ only, starting with a letter or digit)" + ) + # A trailing ~ sorts a version below the same version without it, so a + # Daily always sorts below the Release it will be superseded by. + elif delivery_type == "Daily" and not revision.endswith("~"): + report(f'debian_revision "{revision}" must end in ~ for a Daily entry') + elif delivery_type == "Release" and revision.endswith("~"): + report(f'debian_revision "{revision}" must not end in ~ for a Release entry') + + +def describe(entry, index): + """Label an entry in an error message by what identifies it to a reader.""" + if not isinstance(entry, dict): + return f"entry {index}" + parts = [ + str(entry[field]) + for field in ("kernel_variant", "type", "suite") + if isinstance(entry.get(field), str) + ] + return f"entry {index} ({'/'.join(parts)})" if parts else f"entry {index}" + + +def check_consistency(deliveries, errors): + """Validate the invariants that span entries. + + Entries are written out in full, so the matrix can state a variant twice + and disagree with itself. These checks are what makes that duplication + safe to read at face value. + """ + by_leg = defaultdict(list) + by_variant = defaultdict(list) + by_variant_type = defaultdict(list) + by_package_version = defaultdict(list) + variants_by_package = defaultdict(set) + + for entry in deliveries: + if not isinstance(entry, dict): + continue + variant = entry.get("kernel_variant") + delivery_type = entry.get("type") + suite = entry.get("suite") + if not isinstance(variant, str): + continue + + by_variant[variant].append(entry) + by_variant_type[(variant, delivery_type)].append(entry) + by_leg[(variant, delivery_type, suite)].append(entry) + + for field in ("srcpkg", "binpkg"): + if isinstance(entry.get(field), str) and entry[field]: + variants_by_package[(field, entry[field])].add(variant) + + if isinstance(entry.get("srcpkg"), str) and isinstance( + entry.get("debian_revision"), str + ): + by_package_version[(entry["srcpkg"], entry["debian_revision"])].append(entry) + + for (variant, delivery_type, suite), entries in sorted( + by_leg.items(), key=lambda item: str(item[0]) + ): + if len(entries) > 1: + errors.append( + f"kernel_variant {variant} defines {len(entries)} {delivery_type} " + f"entries for suite {suite}; one entry is one generated package" + ) + + for variant, entries in sorted(by_variant.items()): + for field in VARIANT_IDENTITY_FIELDS: + values = {json.dumps(entry.get(field), sort_keys=True) for entry in entries} + if len(values) > 1: + errors.append( + f"kernel_variant {variant} must use one {field} across all its " + "entries (got " + ", ".join(sorted(values)) + ")" + ) + types = {entry.get("type") for entry in entries} + for delivery_type in DELIVERY_TYPES: + if delivery_type not in types: + errors.append( + f"kernel_variant {variant} has no {delivery_type} entry; " + "every variant must define at least one of each" + ) + + for (variant, delivery_type), entries in sorted( + by_variant_type.items(), key=lambda item: str(item[0]) + ): + for field in REF_FIELDS: + values = {entry.get(field) for entry in entries} + if len(values) > 1: + rendered = ", ".join(sorted(str(v) for v in values)) + errors.append( + f"kernel_variant {variant} must build one {field} across its " + f"{delivery_type} entries (got {rendered})" + ) + + for (field, package), variants in sorted(variants_by_package.items()): + if len(variants) > 1: + errors.append( + f"{field} {package} is shared by kernel variants " + + ", ".join(sorted(variants)) + ) + + for (srcpkg, revision), entries in sorted( + by_package_version.items(), key=lambda item: str(item[0]) + ): + if len(entries) > 1: + suites = ", ".join(sorted(str(entry.get("suite")) for entry in entries)) + errors.append( + f"srcpkg {srcpkg} is built at debian_revision {revision} for " + f"suites {suites}; each entry needs a revision of its own" + ) + + # A suite's Daily and Release differ only by the Daily's trailing ~, so the + # Daily reliably sorts below the Release that supersedes it. + for variant, entries in sorted(by_variant.items()): + revisions = { + (entry.get("type"), entry.get("suite")): entry.get("debian_revision") + for entry in entries + } + for (delivery_type, suite), revision in sorted( + revisions.items(), key=lambda item: str(item[0]) + ): + if delivery_type != "Daily" or not isinstance(revision, str): + continue + release = revisions.get(("Release", suite)) + if isinstance(release, str) and revision != release + "~": + errors.append( + f"kernel_variant {variant} suite {suite}: Daily " + f'debian_revision "{revision}" must be the Release revision ' + f'"{release}" with a trailing ~' + ) + + +def validate(deliveries): + """Return every problem found in the matrix, as a list of messages.""" + errors = [] + for index, entry in enumerate(deliveries): + if not isinstance(entry, dict): + errors.append(f"{describe(entry, index)}: delivery entries must be mappings") + continue + label = describe(entry, index) + check_entry(entry, lambda message, label=label: errors.append(f"{label}: {message}")) + check_consistency(deliveries, errors) + return errors + + +def load_matrix(path): + """Read, parse, and validate the matrix, returning its deliveries.""" + try: + with open(path, encoding="utf-8") as handle: + document = yaml.safe_load(handle) + except FileNotFoundError: + sys.exit(f"ERROR: Matrix file not found: {path}") + except OSError as error: + sys.exit(f"ERROR: Cannot read {path}: {error}") + except yaml.YAMLError as error: + sys.exit(f"ERROR: Invalid YAML in {path}: {error}") + + if not isinstance(document, dict): + sys.exit(f"ERROR: {path}: matrix root must be a mapping with a deliveries key") + + # deliveries is the whole schema. Anything else at the root is a leftover + # from an older matrix (suite_suffix_mapping, say) that would otherwise sit + # there looking authoritative while nothing read it. + unknown_root = sorted(set(document) - {"deliveries"}) + if unknown_root: + sys.exit( + f"ERROR: {path}: unknown top-level key(s) {', '.join(unknown_root)}; " + "deliveries is the only one" + ) + + deliveries = document.get("deliveries") + if not isinstance(deliveries, list): + sys.exit(f"ERROR: {path}: deliveries must be a list") + if not deliveries: + sys.exit(f"ERROR: {path}: deliveries must contain at least one entry") + + errors = validate(deliveries) + if errors: + sys.exit( + f"ERROR: Invalid kernel delivery matrix in {path}:\n" + + "\n".join(f" - {error}" for error in errors) + ) + + return deliveries + + +def select(deliveries, delivery_type, kernel_variant, suite): + """Return the entries matching the requested filters, in matrix order.""" + return [ + entry + for entry in deliveries + if entry["type"] == delivery_type + and kernel_variant in ("", entry["kernel_variant"]) + and suite in ("", entry["suite"]) + ] + + +def describe_selection(delivery_type, kernel_variant, suite): + """Render the active filters for an error message.""" + parts = [f"type={delivery_type}"] + if kernel_variant: + parts.append(f"kernel_variant={kernel_variant}") + if suite: + parts.append(f"suite={suite}") + return " ".join(parts) + + +def for_workflow(entry): + """Shape one entry the way build-kernel-deb.yml's inputs expect it.""" + return { + **entry, + "kernel_config": ",".join(entry["kernel_config"]), + "dkms": ",".join(entry["dkms"]), + } + + +def main(): + parser = argparse.ArgumentParser( + description="Validate and select entries from the kernel delivery matrix.", + epilog="See the module docstring in this file for full documentation.", + ) + parser.add_argument( + "--type", required=True, choices=DELIVERY_TYPES, help="delivery type to select" + ) + parser.add_argument( + "--kernel-variant", default="", help="select only this kernel variant" + ) + parser.add_argument("--suite", default="", help="select only this suite") + parser.add_argument( + "--field", + default="", + help="print just this field of the single selected entry", + ) + parser.add_argument( + "--matrix-file", default=DEFAULT_MATRIX_FILE, help="path to the matrix YAML" + ) + args = parser.parse_args() + + deliveries = load_matrix(args.matrix_file) + selected = select(deliveries, args.type, args.kernel_variant, args.suite) + + what = describe_selection(args.type, args.kernel_variant, args.suite) + if not selected: + sys.exit(f"ERROR: No matrix entries found for {what}") + + if not args.field: + print(json.dumps([for_workflow(entry) for entry in selected], separators=(",", ":"))) + return + + if len(selected) > 1: + sys.exit( + f"ERROR: --field {args.field} needs exactly one entry, but {what} " + f"selects {len(selected)}; narrow it with --kernel-variant and --suite" + ) + + entry = for_workflow(selected[0]) + if args.field not in entry: + sys.exit( + f"ERROR: {what} has no field {args.field}; " + "available: " + ", ".join(sorted(entry)) + ) + print(entry[args.field]) + + +if __name__ == "__main__": + main() diff --git a/ci/scripts/resolve-matrix.sh b/ci/scripts/resolve-matrix.sh deleted file mode 100755 index 4c120641..00000000 --- a/ci/scripts/resolve-matrix.sh +++ /dev/null @@ -1,400 +0,0 @@ -#!/bin/bash -# Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. -# SPDX-License-Identifier: BSD-3-Clause-Clear -set -euo pipefail - -# Resolve and flatten the kernel delivery matrix for a given delivery type. -# -# The matrix root is an object with two top-level keys: -# - "suite_suffix_mapping": a suite -> Debian suffix map shared by every -# kernel variant and delivery type (e.g. "trixie": "~bpo13+1"). -# - "deliveries": the matrix rows. Each kernel_variant owns exactly one -# Daily row and one Release row. A row declares every input needed by -# that delivery, including a debian_version_stub. Three fields are -# list-valued: suites, which is expanded into isolated legs; -# kernel_config, which is one config fragment per element; and dkms, -# which is one out-of-tree module per element. A fragment is -# either a bare name from debian/config-available/ or an "intree:" entry -# naming a path relative to the kernel source root -# (e.g. intree:arch/arm64/configs/qcom_debug.config). A dkms entry is the -# stem of a -dkms package, e.g. "kgsl". -# -# Each flattened leg's final debian_revision is derived from -# debian_version_stub, suite_suffix_mapping[suite], and the delivery type via -# ci/scripts/derive-debian-revision.sh, so the formula has exactly one -# implementation shared with build-kernel-deb.yml's direct-dispatch path. Each -# row also carries debian_version_suffix ("~" for Daily, "" for Release) as a -# visible, validated record of that same delivery-type mapping; it is checked -# against the row's type but never fed into derivation, so a copy/paste error -# here fails fast instead of silently drifting from the formula's single -# implementation. -# -# Usage: -# ci/scripts/resolve-matrix.sh --type Daily -# ci/scripts/resolve-matrix.sh --type Release -# ci/scripts/resolve-matrix.sh --type Daily --single-suite trixie -# ci/scripts/resolve-matrix.sh --type Daily --kernel-variant qcom-next -# ci/scripts/resolve-matrix.sh --type Daily --matrix-file path/to/matrix.json -# -# Options: -# --type TYPE Delivery type to filter (Daily or Release). -# Required. -# --single-suite SUITE Emit only entries for this suite. -# --kernel-variant VARIANT Emit only entries for this kernel variant. -# --matrix-file FILE Path to the matrix JSON file -# (default: ci/build-matrix.json relative to CWD). -# -# Output: -# Compact JSON array to stdout. Every entry has a single suite, the -# kernel_variant that scopes its artifacts, Debusine workspace, and logs, -# and a suite-specific debian_revision (debian_version_stub and -# debian_version_suffix are consumed and removed). kernel_config and dkms are -# joined into the comma-separated strings that build-kernel-deb.yml's -# kernel-config and dkms inputs — and in turn prepare-source.sh's -# --kernel-config and --dkms — expect. -# -# Exit codes: -# 0 Success, at least one entry emitted. -# 1 Error (invalid arguments, matrix validation failure, no matching -# entry, revision derivation failure). - -TYPE="" -SINGLE_SUITE="" -KERNEL_VARIANT="" -MATRIX_FILE="ci/build-matrix.json" - -usage() { - sed -n '/^# Usage:/,/^$/p' "$0" | sed 's/^# \?//' - exit 1 -} - -while [[ $# -gt 0 ]]; do - case $1 in - --type) TYPE="$2"; shift 2 ;; - --single-suite) SINGLE_SUITE="$2"; shift 2 ;; - --kernel-variant) KERNEL_VARIANT="$2"; shift 2 ;; - --matrix-file) MATRIX_FILE="$2"; shift 2 ;; - -h|--help) usage ;; - *) echo "ERROR: Unknown option: $1" >&2; usage ;; - esac -done - -[[ "$TYPE" == "Daily" || "$TYPE" == "Release" ]] || { - echo "ERROR: --type must be Daily or Release" >&2 - exit 1 -} -[[ -f "$MATRIX_FILE" ]] || { echo "ERROR: Matrix file not found: $MATRIX_FILE" >&2; exit 1; } - -jq empty "$MATRIX_FILE" 2>/dev/null \ - || { echo "ERROR: Invalid JSON in $MATRIX_FILE" >&2; exit 1; } - -validation_errors=$(jq -r ' - def required_string($field): - if (has($field) and (.[$field] | type == "string") and (.[$field] | length > 0)) - then empty - else "missing or invalid " + $field - end; - - def optional_string($field): - if (has($field) | not) or (.[$field] | type == "string") - then empty - else "invalid " + $field - end; - - def variant_name_valid: - if (.kernel_variant | type) != "string" - then empty - elif (.kernel_variant | test("^[a-z0-9]+(?:[a-z0-9-]*[a-z0-9])?$")) - then empty - else "kernel_variant must use lowercase letters, digits, and internal hyphens" - end; - - # An "intree:" entry names a fragment shipped by the kernel source, as a - # path relative to the kernel source root (e.g. - # intree:arch/arm64/configs/qcom_debug.config). Reject absolute paths, - # ".." traversal, and anything not ending in .config so a typo fails here - # rather than deep inside prepare-source.sh. - def intree_path_valid: - .[7:] as $path | - ($path | test("^([A-Za-z0-9_.-]+/)*[A-Za-z0-9_.-]+\\.config$")) - and (($path | test("(^|/)\\.\\.(/|$)")) | not); - - # Every fragment lands in debian/config/ under its basename, so two entries - # sharing one basename (e.g. arch/arm64/configs/hardening.config and - # kernel/configs/hardening.config) collide there. - def fragment_basename: - sub("^intree:"; "") | sub("^.*/"; "") | sub("\\.config$"; ""); - - def kernel_config_valid: - if (.kernel_config | type) != "array" - then "kernel_config must be an array" - elif any(.kernel_config[]; type != "string" or length == 0) - then "kernel_config must contain only non-empty strings" - elif any(.kernel_config[]; test(",")) - then "kernel_config entries must not contain commas; use one array element per fragment" - elif ([.kernel_config[]] | unique | length) != (.kernel_config | length) - then "kernel_config must not contain duplicates" - elif any(.kernel_config[]; startswith("intree:") and (intree_path_valid | not)) - then "intree: entries must be a kernel-source-relative path ending in .config (e.g. intree:arch/arm64/configs/qcom_debug.config)" - elif ([.kernel_config[] | fragment_basename] | unique | length) != (.kernel_config | length) - then "kernel_config entries must not resolve to the same fragment filename" - else empty - end; - - # Out-of-tree modules to build and bundle, each named as the stem of its - # -dkms package. The same shape rules debian/rules prepare enforces, - # applied here so a typo fails before any build job starts rather than in - # the middle of one. - def dkms_valid: - if (.dkms | type) != "array" - then "dkms must be an array" - elif any(.dkms[]; type != "string" or length == 0) - then "dkms must contain only non-empty strings" - elif any(.dkms[]; test(",")) - then "dkms entries must not contain commas; use one array element per module" - elif any(.dkms[]; endswith("-dkms")) - then "dkms entries must omit the -dkms suffix (e.g. kgsl, not kgsl-dkms)" - elif any(.dkms[]; test("^[a-z0-9][a-z0-9+.-]*$") | not) - then "dkms entries must be package name stems (lowercase letters, digits, + . -)" - elif ([.dkms[]] | unique | length) != (.dkms | length) - then "dkms must not contain duplicates" - else empty - end; - - def suites_valid: - if (.suites | type) != "array" or (.suites | length) == 0 - then "suites must be a non-empty array" - elif any(.suites[]; type != "string" or length == 0) - then "suites must contain only non-empty strings" - elif any(.suites[]; test("^[a-z0-9]+(?:[a-z0-9-]*[a-z0-9])?$") | not) - then "suites must use lowercase letters, digits, and internal hyphens" - elif ([.suites[]] | unique | length) != (.suites | length) - then "suites must not contain duplicates" - else empty - end; - - def row_errors($index): - if type != "object" - then "row " + ($index | tostring) + ": matrix entries must be objects" - else - . as $row | - [ - required_string("kernel_variant"), - required_string("type"), - required_string("git_clone"), - required_string("branch_or_tag"), - required_string("ref_strategy"), - required_string("srcpkg"), - required_string("binpkg"), - required_string("debian_version_stub"), - optional_string("debusine_parent_workspace"), - optional_string("localversion"), - optional_string("kver_extra"), - variant_name_valid, - suites_valid, - kernel_config_valid, - dkms_valid, - if (.debian_version_stub | type) == "string" and (.debian_version_stub | test("~$")) - then "debian_version_stub must not end in ~" - else empty end, - if (has("debian_version_suffix") | not) or (.debian_version_suffix | type) != "string" - then "missing or invalid debian_version_suffix" - elif .type == "Daily" and .debian_version_suffix != "~" - then "debian_version_suffix must be \"~\" for Daily rows (got \"" + (.debian_version_suffix | tostring) + "\")" - elif .type == "Release" and .debian_version_suffix != "" - then "debian_version_suffix must be \"\" for Release rows (got \"" + (.debian_version_suffix | tostring) + "\")" - else empty end, - if (.type == "Daily" or .type == "Release") - then empty else "type must be Daily or Release" end, - if (.ref_strategy == "latest_tag" or .ref_strategy == "branch_tip" or .ref_strategy == "pinned_ref") - then empty else "ref_strategy must be latest_tag, branch_tip, or pinned_ref" end, - if .type == "Daily" and (.ref_strategy != "latest_tag" and .ref_strategy != "branch_tip") - then "Daily rows must use ref_strategy=latest_tag or ref_strategy=branch_tip" - elif .type == "Release" and .ref_strategy != "pinned_ref" - then "Release rows must use ref_strategy=pinned_ref" - else empty - end, - if .ref_strategy == "latest_tag" - then required_string("tag_pattern") - elif has("tag_pattern") - then "tag_pattern is only valid with ref_strategy=latest_tag" - else empty - end, - if .type == "Release" - then required_string("target_workspace") - elif has("target_workspace") - then "target_workspace is only valid for Release" - else empty - end - ] | .[] | "row " + ($index | tostring) + " (" + (($row.kernel_variant // "unknown") | tostring) + "): " + . - end; - - if type != "object" - then "matrix root must be an object with suite_suffix_mapping and deliveries" - elif (.deliveries | type) != "array" - then "deliveries must be an array" - elif (.deliveries | length) == 0 - then "deliveries must contain at least one row" - elif (.suite_suffix_mapping | type) != "object" - then "suite_suffix_mapping is missing or not an object" - else - .deliveries as $matrix | - .suite_suffix_mapping as $mapping | - ( - [range(0; ($matrix | length)) as $index | $matrix[$index] | row_errors($index)] - + - [ - [$matrix[] | select(type == "object")] - | group_by(.kernel_variant) - | .[] - | . as $rows - | (($rows[0].kernel_variant // "unknown") | tostring) as $variant - | ([ $rows[].type ] | sort) as $types - | ([ $rows[].srcpkg ] | unique) as $srcpkgs - | ([ $rows[].binpkg ] | unique) as $binpkgs - | ([ $rows[].debian_version_stub ] | unique) as $stubs - | if ($rows | length) != 2 - then "kernel_variant " + $variant + " must define exactly one Daily row and one Release row" - elif $types != ["Daily", "Release"] - then "kernel_variant " + $variant + " must define exactly one Daily row and one Release row" - elif ($srcpkgs | length) != 1 - then "kernel_variant " + $variant + " must use one srcpkg across its Daily and Release rows" - elif ($binpkgs | length) != 1 - then "kernel_variant " + $variant + " must use one binpkg across its Daily and Release rows" - elif ($stubs | length) != 1 - then "kernel_variant " + $variant + " must use one debian_version_stub across its Daily and Release rows" - else empty - end - ] - + - [ - [ - $matrix[] - | select(type == "object") - | select((.kernel_variant | type) == "string") - | select((.srcpkg | type) == "string" and (.srcpkg | length) > 0) - | {package: .srcpkg, kernel_variant: .kernel_variant} - ] - | group_by(.package)[] - | ([.[].kernel_variant] | unique) as $variants - | select($variants | length > 1) - | "srcpkg " + .[0].package + " is shared by kernel variants " + ($variants | join(", ")) - ] - + - [ - [ - $matrix[] - | select(type == "object") - | select((.kernel_variant | type) == "string") - | select((.binpkg | type) == "string" and (.binpkg | length) > 0) - | {package: .binpkg, kernel_variant: .kernel_variant} - ] - | group_by(.package)[] - | ([.[].kernel_variant] | unique) as $variants - | select($variants | length > 1) - | "binpkg " + .[0].package + " is shared by kernel variants " + ($variants | join(", ")) - ] - + - [ - $mapping | to_entries[] | select(.value | type != "string") - | "suite_suffix_mapping[" + .key + "] must be a string" - ] - + - [ - $mapping - | to_entries[] - | select((.value | type == "string") and .value != "" and (.value | test("^~") | not)) - | "suite_suffix_mapping[" + .key + "] must be empty or start with ~ (got \"" + .value + "\")" - ] - + - [ - $mapping - | to_entries - | group_by(.value) - | map(select(length > 1)) - | .[]? - | "suites " + ([.[].key] | join(", ")) + " share the same suffix \"" + .[0].value + "\"" - ] - + - [ - [$matrix[] | select(type == "object") | select((.suites | type) == "array") | .suites[]] - | unique - | .[] as $suite - | select(($mapping | has($suite)) | not) - | "suite " + $suite + " has no suite_suffix_mapping entry" - ] - ) | .[] - end -' "$MATRIX_FILE") - -if [[ -n "$validation_errors" ]]; then - echo "ERROR: Invalid kernel delivery matrix:" >&2 - while IFS= read -r error; do - [[ -n "$error" ]] && echo " - $error" >&2 - done <<< "$validation_errors" - exit 1 -fi - -result=$(jq -c \ - --arg type "$TYPE" \ - --arg single_suite "$SINGLE_SUITE" \ - --arg kernel_variant "$KERNEL_VARIANT" ' - [ - .deliveries[] - | select(.type == $type) - | select($kernel_variant == "" or .kernel_variant == $kernel_variant) - | . as $row - | ( - if $single_suite == "" - then .suites - elif (.suites | index($single_suite)) != null - then [$single_suite] - else [] - end - )[] as $suite - | $row - | del(.suites) - | . + { - "suite": $suite, - "kernel_config": ($row.kernel_config | join(",")), - "dkms": ($row.dkms | join(",")) - } - ] - | if length == 0 - then error( - "no matrix entries found for type=" + $type - + (if $kernel_variant != "" then " kernel_variant=" + $kernel_variant else "" end) - + (if $single_suite != "" then " suite=" + $single_suite else "" end) - ) - else . - end - ' "$MATRIX_FILE") || { - echo "ERROR: Matrix resolution failed for type=$TYPE${KERNEL_VARIANT:+ kernel_variant=$KERNEL_VARIANT}${SINGLE_SUITE:+ suite=$SINGLE_SUITE}" >&2 - exit 1 -} - -# Derive each leg's final debian_revision from debian_version_stub, -# suite_suffix_mapping, and its delivery type. derive-debian-revision.sh is -# the single implementation of the formula; build-kernel-deb.yml's direct -# dispatch path calls the same script for the one-suite, no-matrix case. -script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" - -final="[]" -while IFS= read -r leg; do - suite=$(jq -r '.suite' <<< "$leg") - stub=$(jq -r '.debian_version_stub' <<< "$leg") - delivery_type=$(jq -r '.type' <<< "$leg") - variant=$(jq -r '.kernel_variant' <<< "$leg") - - revision=$("$script_dir/derive-debian-revision.sh" \ - --stub "$stub" --suite "$suite" --delivery-type "$delivery_type" \ - --matrix-file "$MATRIX_FILE") || { - echo "ERROR: Failed to derive Debian revision for kernel_variant=$variant suite=$suite type=$delivery_type" >&2 - exit 1 - } - - leg=$(jq -c --arg rev "$revision" '(. + {debian_revision: $rev}) | del(.debian_version_stub, .debian_version_suffix)' <<< "$leg") - final=$(jq -c --argjson leg "$leg" '. + [$leg]' <<< "$final") -done < <(jq -c '.[]' <<< "$result") - -echo "$final" From 269764600acf0233d1496c32d7f1c3351c8c4052 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 02/59] docs: describe the flattened delivery matrix Bring the READMEs onto the schema the previous commit introduced. The Matrix Model section documented suites, debian_version_stub and debian_version_suffix as fields and derive-debian-revision.sh as the single implementation of the revision formula, none of which exist now. Replace the field table with the flattened one, set out the invariants the resolver enforces across entries, and rewrite the maintenance instructions: adding a variant is now one entry per package it should produce rather than exactly two rows, and adding a suite no longer starts with a suite_suffix_mapping entry. The ordering discussion keeps its Daily-sorts-below-Release argument, which the trailing ~ still carries, but now attributes it to the revisions themselves rather than to a mapping. The example matrix becomes YAML and shows one Daily and one Release entry for the same suite, which is what the pairing rule now means. debian/README.md's pointer to the dkms field follows the file to its new name; that field is still unimplemented and is wired up separately. Signed-off-by: Christopher Obbard --- README.md | 249 +++++++++++++++++++++++++---------------------- debian/README.md | 2 +- 2 files changed, 131 insertions(+), 120 deletions(-) diff --git a/README.md b/README.md index 294e6f48..6086a544 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,7 @@ into LOCALVERSION, so each produces a distinct kernel release distinct versioned image package that can be installed alongside the other. See [docs/version.md](docs/version.md) for how the version strings are composed. -`ci/build-matrix.json` is the source of truth; this table is a summary. +`ci/build-matrix.yaml` is the source of truth; this table is a summary. Two entry points use the same reusable build pipeline: @@ -41,56 +41,53 @@ Two entry points use the same reusable build pipeline: - **Release** uses a pinned matrix ref and promotes successful Debian packages to the selected production Debusine workspace. -The final Production matrix is conceptually: - -```json -{ - "suite_suffix_mapping": { - "trixie": "~bpo13+1", - "forky": "", - "resolute": "~26.04.1" - }, - "deliveries": [ - { - "kernel_variant": "qcom-next", - "type": "Daily", - "suites": ["trixie", "forky", "resolute"], - "git_clone": "https://github.com/qualcomm-linux/kernel", - "branch_or_tag": "qcom-next", - "ref_strategy": "latest_tag", - "tag_pattern": "qcom-next-*", - "srcpkg": "linux-qcom-next", - "binpkg": "linux-image-qcom-next", - "kernel_config": [], - "dkms": ["kgsl", "camx", "iris-vpu"], - "debian_version_stub": "0qli1", - "debian_version_suffix": "~" - }, - { - "kernel_variant": "qcom-next", - "type": "Release", - "suites": ["trixie", "forky"], - "git_clone": "https://github.com/qualcomm-linux/kernel", - "branch_or_tag": "", - "ref_strategy": "pinned_ref", - "srcpkg": "linux-qcom-next", - "binpkg": "linux-image-qcom-next", - "kernel_config": [], - "dkms": ["kgsl", "camx", "iris-vpu"], - "debian_version_stub": "0qli1", - "debian_version_suffix": "", - "target_workspace": "qli" - } - ] -} +One entry in `deliveries` is one generated package: a single `kernel_variant`, +a single `type`, and a single `suite`. Nothing is expanded or derived at +resolve time, so what an entry says is what gets built: + +```yaml +deliveries: + - kernel_variant: qcom-next + type: Daily + suite: trixie + git_clone: https://github.com/qualcomm-linux/kernel + branch_or_tag: qcom-next + ref_strategy: latest_tag + tag_pattern: 'qcom-next-*' + srcpkg: linux-qcom-next + binpkg: linux-image-qcom-next + kernel_config: [] + dkms: + - kgsl + - camx + - iris-vpu + - audioreach + debian_revision: '0qli1~bpo13+1~' + + - kernel_variant: qcom-next + type: Release + suite: trixie + git_clone: https://github.com/qualcomm-linux/kernel + branch_or_tag: + ref_strategy: pinned_ref + srcpkg: linux-qcom-next + binpkg: linux-image-qcom-next + kernel_config: [] + dkms: + - kgsl + - camx + - iris-vpu + - audioreach + debian_revision: '0qli1~bpo13+1' + target_workspace: qli ``` -`suite_suffix_mapping` is matrix-wide policy, not duplicated per row: every -suite referenced by any row's `suites` must have an entry here, and every -delivery for a variant derives its final `debian_revision` as -`debian_version_stub + suite_suffix_mapping[suite] + delivery_suffix`, where -`delivery_suffix` is `~` for Daily and empty for Release. For the values -above: +Entries are written out in full rather than sharing YAML anchors, so each one +can be read, grepped, and changed on its own. `resolve-matrix.py` enforces the +consistency that duplication would otherwise put at risk — see +[Matrix Model](#matrix-model). + +Each entry states its `debian_revision` outright. The configured values are: | Suite | Daily | Release | | --- | --- | --- | @@ -99,19 +96,21 @@ above: | Resolute | `0qli1~26.04.1~` | (not a configured Release suite) | `~` always sorts below the same prefix without it in Debian version -ordering, so Daily always sorts below Release for the same suite and stub. -Ordering across *different* suites depends entirely on the configured -suffixes: with the mapping above, Resolute < Trixie < Forky for the same +ordering, so a suite's Daily always sorts below its Release; `resolve-matrix.py` +requires a Daily revision to be exactly its Release revision plus a trailing +`~`. Ordering across *different* suites depends entirely on the configured +revisions: with the values above, Resolute < Trixie < Forky for the same delivery type, matching a Debian-backports-then-unstable promotion chain. This is a deliberate ordering policy, not an automatic guarantee — adding a -suite means choosing a suffix that sorts where that suite belongs relative to -the others. One nuance to be aware of: because Forky's suffix is empty, its -Daily revision ends immediately after the trailing `~`, so Trixie Daily does -not sort below Forky Daily even though Trixie Release sorts below Forky -Release. This does not affect the supported Release-to-Release upgrade path. +suite means choosing a revision that sorts where that suite belongs relative to +the others. One nuance to be aware of: because Forky's revision carries no +suite component, its Daily revision ends immediately after the trailing `~`, so +Trixie Daily does not sort below Forky Daily even though Trixie Release sorts +below Forky Release. This does not affect the supported Release-to-Release +upgrade path. -`ci/build-matrix.json` is the authoritative configuration. Adding a kernel -variant is a two-row matrix change, not a workflow redesign. +`ci/build-matrix.yaml` is the authoritative configuration. Adding a kernel +variant is a matrix change, not a workflow redesign. ## Workflows @@ -165,29 +164,26 @@ production release controls. ## Matrix Model -`ci/build-matrix.json` is an object with two top-level keys: `deliveries` -(the matrix rows) and `suite_suffix_mapping` (matrix-wide Debian suffix -policy, shared by every variant and delivery type). `ci/scripts/resolve-matrix.sh` -validates the document, requires each `kernel_variant` to have exactly one -`Daily` and one `Release` row in `deliveries`, filters by delivery type, and -flattens each `suites` array into independent suite legs. Each leg carries -its own values for: +`ci/build-matrix.yaml` is a mapping with exactly one top-level key, +`deliveries`. One entry in it is one generated package, so there is no +expansion step: `ci/scripts/resolve-matrix.py` validates the whole document, +selects the entries matching the requested type, variant, and suite, and hands +them to the workflow matrix as they stand. Each entry carries: | Field | Purpose | | --- | --- | | `kernel_variant` | Stable identifier for a separately packaged kernel variant. Lowercase letters, digits, and internal hyphens only. | | `type` | `Daily` or `Release`. | -| `suites` | Suites to flatten into individual build legs. Each must have a `suite_suffix_mapping` entry. | +| `suite` | The one suite this entry builds for. | | `git_clone` | Kernel source repository. | | `branch_or_tag` | Source branch or pinned tag, according to `ref_strategy`. | | `ref_strategy` | `latest_tag`, `branch_tip`, or `pinned_ref`. | | `tag_pattern` | Required only for `latest_tag`; matching tags must end in `-YYYYMMDD`, which determines newest-first ordering. | | `srcpkg` | Debian source package name. | | `binpkg` | Kernel image metapackage name. | -| `kernel_config` | Extra fragments applied on top of `debian/config-available/`, all of which is applied to every build, one per array element. A bare name selects `debian/config-available/.config`; an `intree:` entry names a fragment shipped by the kernel source, as a path relative to the kernel source root (e.g. `intree:arch/arm64/configs/qcom_debug.config`), so it stays versioned with the kernel it targets. Empty for variants that need nothing beyond `config-available/`; today it carries only `intree:` fragments. `resolve-matrix.sh` joins it into the comma-separated `kernel-config` workflow input. | -| `dkms` | Out-of-tree DKMS modules built against this kernel and bundled into its `linux-image` package, one per array element, each named as the stem of its `-dkms` package (e.g. `kgsl`). Empty bundles nothing. A listed module is a presence contract: a build fails rather than shipping an image without it. `resolve-matrix.sh` joins it into the comma-separated `dkms` workflow input. | -| `debian_version_stub` | Base Debian revision, shared by a variant's Daily and Release rows. Must not end in `~`; the suite suffix is derived, not stored here. | -| `debian_version_suffix` | `~` for Daily rows, empty for Release rows. Documents the delivery-type half of the revision formula on the row itself; `resolve-matrix.sh` rejects a row where this disagrees with `type`, but derivation always computes this suffix from `type`, never reads this field. | +| `kernel_config` | Extra fragments applied on top of `debian/config-available/`, all of which is applied to every build, one per list element. A bare name selects `debian/config-available/.config`; an `intree:` entry names a fragment shipped by the kernel source, as a path relative to the kernel source root (e.g. `intree:arch/arm64/configs/qcom_debug.config`), so it stays versioned with the kernel it targets. Empty for variants that need nothing beyond `config-available/`; today it carries only `intree:` fragments. `resolve-matrix.py` joins it into the comma-separated `kernel-config` workflow input. | +| `dkms` | Out-of-tree DKMS modules built and bundled into the image package, one per list element, each named without the `-dkms` suffix (e.g. `kgsl`). An empty list bundles nothing. A listed module is a presence contract: a build fails rather than shipping an image without it. `resolve-matrix.py` joins it into the comma-separated `dkms` workflow input, which reaches `prepare-source.sh --dkms`; see [debian/README.md](debian/README.md) for what the packaging does with it. | +| `debian_revision` | The Debian revision this package is built at, stated outright. Daily revisions end in `~`; Release revisions do not. | | `localversion`, `kver_extra` | Optional version overrides forwarded to packaging. | | `debusine_parent_workspace` | Optional parent workspace override for the variant's CI child workspaces. | | `target_workspace` | Debusine destination for Release entries only. | @@ -195,32 +191,45 @@ its own values for: `dkms` currently has one exception the matrix cannot express: `build-kernel-deb.yml` replaces the resolved list with `kgsl` on Ubuntu-family legs, so `camx` and `iris-vpu` are bundled on Debian suites only. That override is temporary and goes -away once the matrix gains per-suite `dkms` lists. - -`target_workspace` is required for `Release` and rejected for `Daily`. -`tag_pattern` is required for `latest_tag` and rejected for other strategies. -The resolver selects the most recent trailing `YYYYMMDD` date, and rejects -duplicate suites and malformed variant identifiers before any build jobs -start. It also rejects a matrix where any configured suite has no -`suite_suffix_mapping` entry, where two suites share the same suffix, where a -suffix is non-empty and doesn't start with `~`, where a variant's Daily -and Release rows disagree on `debian_version_stub`, or where a row's -`debian_version_suffix` doesn't match what its `type` implies — all before -any build job starts. - -Each flattened leg's final `debian_revision` is derived by -`ci/scripts/derive-debian-revision.sh` from `debian_version_stub`, -`suite_suffix_mapping[suite]`, and the delivery type -(`stub + suffix + "~"` for Daily, `stub + suffix` for Release). This script is -the single implementation of the formula: `resolve-matrix.sh` calls it once -per flattened leg, and `build-kernel-deb.yml`'s direct-dispatch path (which -has no full-matrix context) calls the same script for the one suite it was -given. - -Each leg has a distinct prepared-source artifact, Debusine child workspace, and -S3 path keyed by `kernel_variant + suite`. This prevents two variants that both -build, for example, `trixie` from consuming or publishing each other's inputs -or outputs. +away once the per-suite `dkms` lists the flattened matrix now allows are used. + +`resolve-matrix.py` rejects the matrix — before any build job starts — where an +entry has an unknown field or a missing required one, a malformed variant or +suite identifier, a `ref_strategy` its `type` does not allow (`Daily` must +track something moving, `Release` must be pinned), a `tag_pattern` without +`latest_tag`, a `target_workspace` on a `Daily` entry or none on a `Release` +one, a `kernel_config` fragment that escapes the kernel source root or collides +with another fragment's filename, a `dkms` entry that is not a package name +stem or repeats, or a `debian_revision` that is not a valid Debian revision or +carries the wrong trailing `~` for its `type`. + +Because entries are written out in full, the resolver also checks the +invariants that span them, which is what makes the duplication safe to read at +face value: + +- No two entries share a `kernel_variant`, `type` and `suite` — one entry is + one generated package. +- A variant's entries agree on `srcpkg`, `binpkg`, `kernel_config` and `dkms`; + those decide what the package *is*, and the entries differ only in where it + goes. +- A variant's entries of one `type` agree on `git_clone`, `branch_or_tag`, + `ref_strategy` and `tag_pattern`, so a forgotten suite cannot quietly ship a + different kernel from its siblings after a release ref bump. +- Every variant defines at least one `Daily` and one `Release` entry. +- No `srcpkg` or `binpkg` is shared between variants, and no two entries build + the same `srcpkg` at the same `debian_revision`. +- Where a suite has both, its `Daily` revision is its `Release` revision plus a + trailing `~`. + +`build-kernel-deb.yml`'s direct-dispatch path has no matrix context of its own, +so when its `debian-revision` input is empty it looks up the `Daily` entry for +the variant and suite it was given (`resolve-matrix.py --field +debian_revision`) and builds at the revision the daily build would have used. + +Each entry has a distinct prepared-source artifact, Debusine child workspace, +and S3 path keyed by `kernel_variant + suite`. This prevents two variants that +both build, for example, `trixie` from consuming or publishing each other's +inputs or outputs. Daily S3 outputs use these layouts, where `` is `-`: @@ -237,10 +246,9 @@ Supporting scripts keep workflow YAML small and testable: | Script | Responsibility | | --- | --- | -| `ci/scripts/resolve-matrix.sh` | Validates and flattens matrix rows. | +| `ci/scripts/resolve-matrix.py` | Validates the delivery matrix and selects the entries to build. Needs PyYAML (`python3-yaml`). | | `ci/scripts/resolve-kernel-ref.sh` | Resolves a matrix-selected dated tag or validates a direct ref. | | `ci/scripts/derive-localversion.sh` | Derives the version fields from the variant, resolved kernel ref and HEAD, printing `LOCALVERSION=`, `SNAPSHOT=` and `GITSHA=` lines. `SNAPSHOT` is the dated component of the Debian version: the tag's date, or the HEAD commit date for a branch-tip build. Scheme and rationale: [docs/version.md](docs/version.md). | -| `ci/scripts/derive-debian-revision.sh` | Derives the final suite-specific `debian_revision` from `debian_version_stub`, `suite_suffix_mapping`, and delivery type. | ## Architecture @@ -451,7 +459,8 @@ The available inputs are: | `srcpkg` | `linux-qcom-next` | Advanced source package identity override. | | `binpkg` | `linux-image-qcom-next` | Advanced image metapackage identity override. | | `kernel-config` | Empty | Advanced extra fragments applied on top of all of `debian/config-available/`, e.g. `intree:arch/arm64/configs/qcom_debug.config`. | -| `debian-version-stub` | `0qli1` | Advanced Debian version stub. The selected suite's mapped suffix and a Daily-style trailing `~` are applied automatically; direct builds always use Daily semantics since they are build-only and non-promoting. | +| `dkms` | Empty | Advanced comma-separated DKMS modules bundled into the image package, each without the `-dkms` suffix. Empty bundles none. | +| `debian-revision` | The matrix Daily revision | Advanced Debian revision override. Left empty, the build takes the `debian_revision` of the matrix's `Daily` entry for the selected variant and suite; direct builds always use the Daily entry since they are build-only and non-promoting. | | `localversion` | Auto-derived | Advanced explicit `LOCALVERSION` override. | | `kver-extra` | Empty | Advanced kernel-release suffix. | | `debug-build` | `false` | Advanced debug configuration toggle. | @@ -487,37 +496,39 @@ and keeps production approval controls in the workflow path. To add a kernel variant: -1. Add exactly two rows to `deliveries` with the same `kernel_variant`: one - `Daily` and one `Release`. -2. Define all package identity, source/ref strategy, configuration, - `debian_version_stub`, and suite values in both rows. Do not rely on - another variant's values. `srcpkg`, `binpkg`, and `debian_version_stub` - must remain identical across the pair. Set `debian_version_suffix` to `~` - on the Daily row and `""` on the Release row; `resolve-matrix.sh` rejects - the pair if either disagrees with its row's `type`. +1. Add one entry to `deliveries` per package the variant should produce: one + per Daily suite and one per Release suite, each spelling out all of its own + fields. Do not rely on another variant's values. +2. Keep `srcpkg`, `binpkg` and `kernel_config` identical across every entry for + the variant, and the ref fields identical across its entries of one `type`. + `resolve-matrix.py` rejects the matrix if they drift apart. 3. Use `latest_tag` with a dated tag glob or `branch_tip` for Daily. Use `pinned_ref` for Release, and update that ref through a reviewed PR. -4. Give the variant distinct `srcpkg` and `binpkg` values. Set the Release - `target_workspace` explicitly. -5. Confirm suite-family routing: Debian suites use Debusine; Ubuntu suites use +4. Give the variant distinct `srcpkg` and `binpkg` values. Set + `target_workspace` on each Release entry. +5. Give each entry a `debian_revision`: the Daily one is the Release one for + the same suite plus a trailing `~`. +6. Confirm suite-family routing: Debian suites use Debusine; Ubuntu suites use the Docker path. -6. Run a filtered Daily validation for the new variant, then its full Daily and +7. Run a filtered Daily validation for the new variant, then its full Daily and Release flows. To add a new suite (for an existing or new variant): -1. Add an entry for it to the shared top-level `suite_suffix_mapping`, empty - or starting with `~`, and distinct from every other suite's suffix. -2. Add the suite to the `suites` array of the relevant Daily and/or Release - rows. `resolve-matrix.sh` rejects any configured suite with no mapping - entry before any build job starts. -3. Choose the suffix so the suite sorts where it belongs relative to the +1. Add one entry per delivery type the suite should get, copying the variant's + existing entry for that type and changing `suite` and `debian_revision`. +2. Choose the revision so the suite sorts where it belongs relative to the others for the same delivery type (see the ordering discussion in - [Overview](#overview)). + [Overview](#overview)), and so it does not collide with another entry + building the same `srcpkg`. No workflow dispatch choices need to be updated: manual Daily and Release inputs accept matrix-defined variant and suite strings. +Run `ci/scripts/resolve-matrix.py --type Daily` and `--type Release` locally to +validate a matrix change before pushing it; both validate the whole document, +so either one catches a mistake in the other's entries. + ## Contributing See [CONTRIBUTING.md](CONTRIBUTING.md) for branch, review, and DCO diff --git a/debian/README.md b/debian/README.md index 581753c2..0863f55c 100644 --- a/debian/README.md +++ b/debian/README.md @@ -437,7 +437,7 @@ A listed module is a presence contract — if it fails to build, or its `BUILD_EXCLUSIVE` gates exclude this kernel, the package build fails rather than shipping a kernel image without it. To stop bundling a module, drop it from the list its build was given (for CI builds, the `dkms` field in -`ci/build-matrix.json`). +`ci/build-matrix.yaml`). ### Standalone developer use From 9d3c7911e4c89ae636741be5de1d5f7135e05b7a Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 03/59] ci: move the Ubuntu DKMS exception into the delivery matrix camx-dkms and iris-vpu-dkms are not packaged for Ubuntu, so build-kernel-deb.yml overwrote the resolved DKMS list with kgsl on Ubuntu-family legs. That was delivery policy sitting in workflow YAML, put there because a matrix row spanned several suites and could not say anything about one of them. A row no longer does: one entry is one generated package for one suite, so the resolute entry can carry its own dkms list and does. Revert the override, along with the "needs: resolve" and SUITE_FAMILY plumbing that fed it, and pass the resolved list to prepare-source.sh unchanged. dkms stops being a variant identity field. It was checked alongside srcpkg, binpkg and kernel_config, on the reasoning that those decide what the package is and only the destination varies. That holds for the others, which describe the kernel itself, but not for dkms: it names packages that have to exist in the target archive, so it is a property of the variant in a suite rather than of the variant. A variant's entries for one suite must still agree, which keeps the property worth having - a Daily that bundled a different module set from the Release it precedes would not be testing what ships - while letting the suites differ from each other. What each leg bundles is unchanged: the Debian suites get kgsl, camx and iris-vpu, resolute gets kgsl. It is now stated in the matrix rather than applied to it at build time. Signed-off-by: Christopher Obbard --- .github/workflows/build-kernel-deb.yml | 21 +-------------------- README.md | 16 +++++++--------- ci/build-matrix.yaml | 13 ++++++++----- ci/scripts/resolve-matrix.py | 22 +++++++++++++++++++++- 4 files changed, 37 insertions(+), 35 deletions(-) diff --git a/.github/workflows/build-kernel-deb.yml b/.github/workflows/build-kernel-deb.yml index 58e05bef..db87554a 100644 --- a/.github/workflows/build-kernel-deb.yml +++ b/.github/workflows/build-kernel-deb.yml @@ -194,8 +194,6 @@ jobs: prepare: name: Prepare kernel source runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] - # TEMPORARY, with the DKMS override in "Prepare source" below. - needs: resolve env: KERNEL_URL: ${{ inputs.git-clone || inputs.kernel-url || 'https://github.com/qualcomm-linux/kernel' }} @@ -378,29 +376,12 @@ jobs: KVER_EXTRA_INPUT: ${{ inputs.kver-extra }} KERNEL_CONFIG_INPUT: ${{ inputs.kernel-config }} DKMS_INPUT: ${{ inputs.dkms }} - # TEMPORARY, consumed by the DKMS override below. - SUITE_FAMILY: ${{ needs.resolve.outputs.family }} DEBUG_BUILD_INPUT: ${{ inputs.debug-build }} run: | # No -x here: keep xtrace off for this step as it assembles the # prepare-source.sh argument list which may include sensitive paths. set -euo pipefail - DKMS_LIST="$DKMS_INPUT" - - # TEMPORARY: camx and iris-vpu are bundled on non-Ubuntu builds only. - # A listed module is a presence contract, so an Ubuntu leg that asked - # for them would hard-fail rather than quietly ship without them. - # - # This is delivery policy and belongs in ci/build-matrix.yaml, whose - # entries are now one per suite and can carry it. Revert this - # override — and the "needs: resolve"/SUITE_FAMILY plumbing that - # feeds it — in the commit that moves the policy there. - if [[ "$SUITE_FAMILY" == "ubuntu" ]]; then - DKMS_LIST="kgsl" - echo "Ubuntu build: DKMS list overridden to '$DKMS_LIST' (was '$DKMS_INPUT')" - fi - ARGS=( --source-dir "$KERNEL_DIR" --distro "$DISTRO" @@ -414,7 +395,7 @@ jobs: [[ -n "$KERNEL_CONFIG_INPUT" ]] && ARGS+=(--kernel-config "$KERNEL_CONFIG_INPUT") # Always passed, unlike the optional inputs above: an empty list is a # genuine "bundle nothing" rather than a request for some default. - ARGS+=(--dkms "$DKMS_LIST") + ARGS+=(--dkms "$DKMS_INPUT") [[ "$DEBUG_BUILD_INPUT" == "true" ]] && ARGS+=(--debug) docker run -i --rm \ diff --git a/README.md b/README.md index 6086a544..0ddb7327 100644 --- a/README.md +++ b/README.md @@ -182,17 +182,12 @@ them to the workflow matrix as they stand. Each entry carries: | `srcpkg` | Debian source package name. | | `binpkg` | Kernel image metapackage name. | | `kernel_config` | Extra fragments applied on top of `debian/config-available/`, all of which is applied to every build, one per list element. A bare name selects `debian/config-available/.config`; an `intree:` entry names a fragment shipped by the kernel source, as a path relative to the kernel source root (e.g. `intree:arch/arm64/configs/qcom_debug.config`), so it stays versioned with the kernel it targets. Empty for variants that need nothing beyond `config-available/`; today it carries only `intree:` fragments. `resolve-matrix.py` joins it into the comma-separated `kernel-config` workflow input. | -| `dkms` | Out-of-tree DKMS modules built and bundled into the image package, one per list element, each named without the `-dkms` suffix (e.g. `kgsl`). An empty list bundles nothing. A listed module is a presence contract: a build fails rather than shipping an image without it. `resolve-matrix.py` joins it into the comma-separated `dkms` workflow input, which reaches `prepare-source.sh --dkms`; see [debian/README.md](debian/README.md) for what the packaging does with it. | +| `dkms` | Out-of-tree DKMS modules built and bundled into the image package, one per list element, each named without the `-dkms` suffix (e.g. `kgsl`). Each needs a `-dkms` package in the suite being built for, so this varies between suites. An empty list bundles nothing. A listed module is a presence contract: a build fails rather than shipping an image without it. `resolve-matrix.py` joins it into the comma-separated `dkms` workflow input, which reaches `prepare-source.sh --dkms`; see [debian/README.md](debian/README.md) for what the packaging does with it. | | `debian_revision` | The Debian revision this package is built at, stated outright. Daily revisions end in `~`; Release revisions do not. | | `localversion`, `kver_extra` | Optional version overrides forwarded to packaging. | | `debusine_parent_workspace` | Optional parent workspace override for the variant's CI child workspaces. | | `target_workspace` | Debusine destination for Release entries only. | -`dkms` currently has one exception the matrix cannot express: `build-kernel-deb.yml` -replaces the resolved list with `kgsl` on Ubuntu-family legs, so `camx` and -`iris-vpu` are bundled on Debian suites only. That override is temporary and goes -away once the per-suite `dkms` lists the flattened matrix now allows are used. - `resolve-matrix.py` rejects the matrix — before any build job starts — where an entry has an unknown field or a missing required one, a malformed variant or suite identifier, a `ref_strategy` its `type` does not allow (`Daily` must @@ -209,9 +204,12 @@ face value: - No two entries share a `kernel_variant`, `type` and `suite` — one entry is one generated package. -- A variant's entries agree on `srcpkg`, `binpkg`, `kernel_config` and `dkms`; - those decide what the package *is*, and the entries differ only in where it - goes. +- A variant's entries agree on `srcpkg`, `binpkg` and `kernel_config`; those + decide what the package *is*, and the entries differ only in where it goes. +- A variant's entries for one suite agree on `dkms`. The module set depends on + which `-dkms` packages the target archive carries, so it varies between + suites — but a suite's Daily and Release must match, or the Daily is not + testing the module set the Release will ship. - A variant's entries of one `type` agree on `git_clone`, `branch_or_tag`, `ref_strategy` and `tag_pattern`, so a forgotten suite cannot quietly ship a different kernel from its siblings after a release ref bump. diff --git a/ci/build-matrix.yaml b/ci/build-matrix.yaml index 208beb61..86df3c28 100644 --- a/ci/build-matrix.yaml +++ b/ci/build-matrix.yaml @@ -13,8 +13,13 @@ # Entries are written out in full rather than sharing YAML anchors, so each one # can be read, grepped, and changed on its own. resolve-matrix.py enforces the # consistency that duplication would otherwise put at risk: a variant's entries -# must agree on srcpkg, binpkg and kernel_config, and its entries of one -# delivery type must agree on the kernel ref they build. +# must agree on srcpkg, binpkg and kernel_config, its entries of one delivery +# type must agree on the kernel ref they build, and its entries for one suite +# must agree on dkms. +# +# dkms varies by suite because it depends on which -dkms packages the +# target archive carries: the Debian suites bundle camx and iris-vpu, which +# Ubuntu does not package. # # debian_revision is stated outright, not derived. Daily revisions end in "~" # so they sort below the Release revision for the same suite; the rest of the @@ -70,11 +75,9 @@ deliveries: srcpkg: linux-qcom-next binpkg: linux-image-qcom-next kernel_config: [] + # kgsl alone: camx-dkms and iris-vpu-dkms are not packaged for Ubuntu. dkms: - kgsl - - camx - - iris-vpu - - audioreach debian_revision: '0qli1~26.04.1~' - kernel_variant: qcom-next diff --git a/ci/scripts/resolve-matrix.py b/ci/scripts/resolve-matrix.py index 4f97703c..f92342cf 100755 --- a/ci/scripts/resolve-matrix.py +++ b/ci/scripts/resolve-matrix.py @@ -100,7 +100,14 @@ # Fields that identify the variant itself rather than one of its build legs. # Every entry for a variant must agree on them, because they decide what the # package is; the entries only differ in where it is delivered. -VARIANT_IDENTITY_FIELDS = ("srcpkg", "binpkg", "kernel_config", "dkms") +VARIANT_IDENTITY_FIELDS = ("srcpkg", "binpkg", "kernel_config") + +# Fields a variant may vary between suites but not within one. The out-of-tree +# module set depends on which -dkms packages the target archive has, so +# it is a property of the variant in a suite rather than of the variant. A +# suite's Daily and Release must still agree: a Daily that bundles a different +# module set from the Release it precedes is not testing what will ship. +SUITE_IDENTITY_FIELDS = ("dkms",) # Fields deciding which kernel tree is built. All entries for one variant and # delivery type build the same source, so a stale suite cannot quietly ship a @@ -289,6 +296,7 @@ def check_consistency(deliveries, errors): by_variant_type = defaultdict(list) by_package_version = defaultdict(list) variants_by_package = defaultdict(set) + by_variant_suite = defaultdict(list) for entry in deliveries: if not isinstance(entry, dict): @@ -301,6 +309,7 @@ def check_consistency(deliveries, errors): by_variant[variant].append(entry) by_variant_type[(variant, delivery_type)].append(entry) + by_variant_suite[(variant, suite)].append(entry) by_leg[(variant, delivery_type, suite)].append(entry) for field in ("srcpkg", "binpkg"): @@ -337,6 +346,17 @@ def check_consistency(deliveries, errors): "every variant must define at least one of each" ) + for (variant, suite), entries in sorted( + by_variant_suite.items(), key=lambda item: str(item[0]) + ): + for field in SUITE_IDENTITY_FIELDS: + values = {json.dumps(entry.get(field), sort_keys=True) for entry in entries} + if len(values) > 1: + errors.append( + f"kernel_variant {variant} must use one {field} across its " + f"{suite} entries (got " + ", ".join(sorted(values)) + ")" + ) + for (variant, delivery_type), entries in sorted( by_variant_type.items(), key=lambda item: str(item[0]) ): From 68ca93792706a111e6d65654d3a6513d95c0cbbe Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 04/59] ci: stop the build job name nesting variant above suite The matrix build job was named " / ". GitHub composes a called workflow's job names as " / " and renders the result as a tree, so the separator inside the caller's own name added a level that means nothing: every run grew a qcom-next hub and a qcom-next-debug hub, each holding its suites, when the thing being run is one leg per suite. Name the job " ()" instead, which is how GitHub writes its own matrix job names, so a leg reads as one entry. The jobs themselves are unchanged. The remaining nesting under each leg is the reusable workflow's own jobs, which GitHub always surfaces individually. Signed-off-by: Christopher Obbard --- .github/workflows/daily.yml | 2 +- .github/workflows/pr-build.yml | 2 +- .github/workflows/release.yml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/daily.yml b/.github/workflows/daily.yml index 1e30522c..f6b85be0 100644 --- a/.github/workflows/daily.yml +++ b/.github/workflows/daily.yml @@ -109,7 +109,7 @@ jobs: # build does not cancel the forky build. # --------------------------------------------------------------------------- build: - name: ${{ matrix.kernel_variant }} / ${{ matrix.suite }} + name: ${{ matrix.kernel_variant }} (${{ matrix.suite }}) needs: configure-matrix strategy: fail-fast: false diff --git a/.github/workflows/pr-build.yml b/.github/workflows/pr-build.yml index 2d2888ae..dd964214 100644 --- a/.github/workflows/pr-build.yml +++ b/.github/workflows/pr-build.yml @@ -52,7 +52,7 @@ jobs: # resolute results on the PR. # --------------------------------------------------------------------------- build: - name: ${{ matrix.kernel_variant }} / ${{ matrix.suite }} + name: ${{ matrix.kernel_variant }} (${{ matrix.suite }}) needs: configure-matrix strategy: fail-fast: false diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 62e31eea..cc7693df 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -93,7 +93,7 @@ jobs: # release path (lib/release) instead of the S3 publish path. # --------------------------------------------------------------------------- build: - name: ${{ matrix.kernel_variant }} / ${{ matrix.suite }} + name: ${{ matrix.kernel_variant }} (${{ matrix.suite }}) needs: configure-matrix strategy: fail-fast: false From 3c4c0903271e040956eeeba23da670850f9869a3 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 05/59] ci: keep kernel_variant out of the kernel version kernel_variant was doing two unrelated jobs. It named the build in CI - the Actions job, the prepared-source artifact, the Debusine child workspace, the S3 path - and it was also handed to derive-localversion.sh, which folded it into LOCALVERSION and so into the kernel release string and the versioned linux-image package name. Renaming a build leg therefore renamed the kernel that leg installs, which is not something a CI label should be able to do. Split the two. Add a flavour field: the kernel's own identity, the part of the release string that distinguishes two kernels built from one ref with different configuration so their linux-image packages coexist. derive-localversion.sh takes --flavour in place of --variant, and build-kernel-deb.yml feeds it from the new flavour input rather than kernel-variant. kernel_variant keeps every CI-facing use and reaches nothing that ends up in a package. Everything the matrix asserts about what is built is regrouped onto flavour, since that is now what identifies a package: the srcpkg, binpkg and kernel_config agreement, the per-suite dkms agreement, the kernel ref agreement within a delivery type, the Daily-and-Release requirement, the srcpkg and binpkg exclusivity, and the Daily/Release revision pairing. kernel_variant keeps one check of its own, that it does not repeat for a type and suite, which is what makes job names and artifact scopes unique. The flavours are qcom-next and qcom-next-debug, the values kernel_variant already held, so every leg builds the same kernel release it did before: verified leg by leg against the old script, LOCALVERSION identical for all nine. kernel_variant is now free. Renaming the legs to qcom-next-trixie and so on validates cleanly and leaves LOCALVERSION untouched, which it could not have done before this change. Signed-off-by: Christopher Obbard --- .github/workflows/build-kernel-deb.yml | 17 +++-- .github/workflows/daily.yml | 6 +- .github/workflows/pr-build.yml | 1 + .github/workflows/release.yml | 1 + README.md | 38 ++++++----- ci/build-matrix.yaml | 19 +++++- ci/scripts/derive-localversion.sh | 36 ++++++----- ci/scripts/resolve-matrix.py | 88 +++++++++++++++----------- 8 files changed, 133 insertions(+), 73 deletions(-) diff --git a/.github/workflows/build-kernel-deb.yml b/.github/workflows/build-kernel-deb.yml index db87554a..f620d365 100644 --- a/.github/workflows/build-kernel-deb.yml +++ b/.github/workflows/build-kernel-deb.yml @@ -4,7 +4,12 @@ on: workflow_dispatch: inputs: kernel-variant: - description: 'Kernel variant identifier' + description: 'Kernel variant identifier, naming this build in CI only' + required: true + default: 'qcom-next' + type: string + flavour: + description: 'Kernel flavour: the LOCALVERSION suffix and so the kernel release identity' required: true default: 'qcom-next' type: string @@ -82,7 +87,11 @@ on: workflow_call: inputs: kernel-variant: - description: 'Kernel variant identifier' + description: 'Kernel variant identifier, naming this build in CI only' + type: string + default: 'qcom-next' + flavour: + description: 'Kernel flavour: the LOCALVERSION suffix and so the kernel release identity' type: string default: 'qcom-next' suite: @@ -278,7 +287,7 @@ jobs: - name: Derive LOCALVERSION env: LOCALVERSION_INPUT: ${{ inputs.localversion }} - KERNEL_VARIANT_INPUT: ${{ inputs.kernel-variant || 'qcom-next' }} + FLAVOUR_INPUT: ${{ inputs.flavour || 'qcom-next' }} run: | set -euo pipefail OVERRIDE_LV="$LOCALVERSION_INPUT" @@ -287,7 +296,7 @@ jobs: echo "LOCALVERSION=$OVERRIDE_LV" >> "$GITHUB_ENV" else LOCALVERSION=$(ci/scripts/derive-localversion.sh \ - --variant "$KERNEL_VARIANT_INPUT" \ + --flavour "$FLAVOUR_INPUT" \ --ref "$KERNEL_REF" \ --sha "$(echo "$KERNEL_SHA" | cut -c1-12)") echo "LOCALVERSION=$LOCALVERSION" >> "$GITHUB_ENV" diff --git a/.github/workflows/daily.yml b/.github/workflows/daily.yml index f6b85be0..e5d7cf4b 100644 --- a/.github/workflows/daily.yml +++ b/.github/workflows/daily.yml @@ -9,9 +9,10 @@ name: daily # Matrix schema: ci/build-matrix.yaml # One entry is one generated package, so ci/scripts/resolve-matrix.py selects # the entries with type: Daily and passes them through as they stand. Every -# field (kernel_variant, suite, srcpkg, binpkg, kernel_config, dkms, +# field (kernel_variant, suite, flavour, srcpkg, binpkg, kernel_config, dkms, # debian_revision, git_clone, branch_or_tag, ref_strategy) becomes a -# build-kernel-deb.yml input. +# build-kernel-deb.yml input. kernel_variant names the job and scopes its +# artifacts; flavour is what reaches the built kernel's version. # # To add a daily build target: add an entry to ci/build-matrix.yaml with # type: Daily. No workflow changes needed. @@ -119,6 +120,7 @@ jobs: with: suite: ${{ matrix.suite }} kernel-variant: ${{ matrix.kernel_variant }} + flavour: ${{ matrix.flavour }} git-clone: ${{ matrix.git_clone }} kernel-branch: ${{ matrix.branch_or_tag }} ref-strategy: ${{ matrix.ref_strategy }} diff --git a/.github/workflows/pr-build.yml b/.github/workflows/pr-build.yml index dd964214..4ef4263a 100644 --- a/.github/workflows/pr-build.yml +++ b/.github/workflows/pr-build.yml @@ -62,6 +62,7 @@ jobs: with: suite: ${{ matrix.suite }} kernel-variant: ${{ matrix.kernel_variant }} + flavour: ${{ matrix.flavour }} git-clone: ${{ matrix.git_clone }} kernel-branch: ${{ matrix.branch_or_tag }} ref-strategy: ${{ matrix.ref_strategy }} diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index cc7693df..80707bc6 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -103,6 +103,7 @@ jobs: with: suite: ${{ matrix.suite }} kernel-variant: ${{ matrix.kernel_variant }} + flavour: ${{ matrix.flavour }} git-clone: ${{ matrix.git_clone }} kernel-branch: ${{ matrix.branch_or_tag }} ref-strategy: ${{ matrix.ref_strategy }} diff --git a/README.md b/README.md index 0ddb7327..4c7b452a 100644 --- a/README.md +++ b/README.md @@ -15,9 +15,8 @@ kernel variants, each with independent source/package identity, kernel source and ref strategy, configuration fragments, Debian revision, target suites, and release destination. -Every kernel variant owns exactly two complete matrix rows: one `Daily` row and -one `Release` row. The resolver expands every suite in those rows into an -isolated `kernel_variant + suite` build leg. +Every entry in the matrix is one isolated build leg producing one package. A +kernel flavour owns as many entries as it has suites and delivery types. ### Configured variants @@ -26,11 +25,13 @@ isolated `kernel_variant + suite` build leg. | `qcom-next` | `linux-qcom-next` | `linux-image-qcom-next` | trixie, forky, resolute | trixie, forky | Standard kernel | | `qcom-next-debug` | `linux-qcom-next-debug` | `linux-image-qcom-next-debug` | trixie, forky | trixie, forky | Adds `arch/arm64/configs/qcom_debug.config` and `kernel/configs/debug.config` from the kernel source, via `intree:` entries | -Both build the same kernel ref. `derive-localversion.sh` folds the variant name +Both build the same kernel ref. `derive-localversion.sh` folds the *flavour* into LOCALVERSION, so each produces a distinct kernel release (`+qcom-next--g` and `+qcom-next-debug--g`) and therefore a distinct versioned image package that can be installed alongside the other. See [docs/version.md](docs/version.md) for how the version strings are composed. +The flavour is what the kernel is; `kernel_variant` is only what CI calls the +build. `ci/build-matrix.yaml` is the source of truth; this table is a summary. @@ -50,6 +51,7 @@ deliveries: - kernel_variant: qcom-next type: Daily suite: trixie + flavour: qcom-next git_clone: https://github.com/qualcomm-linux/kernel branch_or_tag: qcom-next ref_strategy: latest_tag @@ -67,6 +69,7 @@ deliveries: - kernel_variant: qcom-next type: Release suite: trixie + flavour: qcom-next git_clone: https://github.com/qualcomm-linux/kernel branch_or_tag: ref_strategy: pinned_ref @@ -172,9 +175,10 @@ them to the workflow matrix as they stand. Each entry carries: | Field | Purpose | | --- | --- | -| `kernel_variant` | Stable identifier for a separately packaged kernel variant. Lowercase letters, digits, and internal hyphens only. | +| `kernel_variant` | CI identifier for this build: its Actions job name, prepared-source artifact, Debusine child workspace and S3 path. Never reaches a package name or version. Lowercase letters, digits, and internal hyphens only. | | `type` | `Daily` or `Release`. | | `suite` | The one suite this entry builds for. | +| `flavour` | The kernel's own identity, and the only matrix field that reaches the built kernel. `derive-localversion.sh` makes it the LOCALVERSION suffix, so two flavours built from one ref get distinct kernel releases and their `linux-image` packages coexist. All entries sharing a flavour build the same package for different suites. | | `git_clone` | Kernel source repository. | | `branch_or_tag` | Source branch or pinned tag, according to `ref_strategy`. | | `ref_strategy` | `latest_tag`, `branch_tip`, or `pinned_ref`. | @@ -204,17 +208,17 @@ face value: - No two entries share a `kernel_variant`, `type` and `suite` — one entry is one generated package. -- A variant's entries agree on `srcpkg`, `binpkg` and `kernel_config`; those +- A flavour's entries agree on `srcpkg`, `binpkg` and `kernel_config`; those decide what the package *is*, and the entries differ only in where it goes. -- A variant's entries for one suite agree on `dkms`. The module set depends on +- A flavour's entries for one suite agree on `dkms`. The module set depends on which `-dkms` packages the target archive carries, so it varies between suites — but a suite's Daily and Release must match, or the Daily is not testing the module set the Release will ship. -- A variant's entries of one `type` agree on `git_clone`, `branch_or_tag`, +- A flavour's entries of one `type` agree on `git_clone`, `branch_or_tag`, `ref_strategy` and `tag_pattern`, so a forgotten suite cannot quietly ship a different kernel from its siblings after a release ref bump. -- Every variant defines at least one `Daily` and one `Release` entry. -- No `srcpkg` or `binpkg` is shared between variants, and no two entries build +- Every flavour defines at least one `Daily` and one `Release` entry. +- No `srcpkg` or `binpkg` is shared between flavours, and no two entries build the same `srcpkg` at the same `debian_revision`. - Where a suite has both, its `Daily` revision is its `Release` revision plus a trailing `~`. @@ -246,7 +250,7 @@ Supporting scripts keep workflow YAML small and testable: | --- | --- | | `ci/scripts/resolve-matrix.py` | Validates the delivery matrix and selects the entries to build. Needs PyYAML (`python3-yaml`). | | `ci/scripts/resolve-kernel-ref.sh` | Resolves a matrix-selected dated tag or validates a direct ref. | -| `ci/scripts/derive-localversion.sh` | Derives the version fields from the variant, resolved kernel ref and HEAD, printing `LOCALVERSION=`, `SNAPSHOT=` and `GITSHA=` lines. `SNAPSHOT` is the dated component of the Debian version: the tag's date, or the HEAD commit date for a branch-tip build. Scheme and rationale: [docs/version.md](docs/version.md). | +| `ci/scripts/derive-localversion.sh` | Derives the version fields from the flavour, resolved kernel ref and HEAD, printing `LOCALVERSION=`, `SNAPSHOT=` and `GITSHA=` lines. `SNAPSHOT` is the dated component of the Debian version: the tag's date, or the HEAD commit date for a branch-tip build. Scheme and rationale: [docs/version.md](docs/version.md). | ## Architecture @@ -448,7 +452,8 @@ The available inputs are: | Input | Default | Purpose | | --- | --- | --- | -| `kernel-variant` | `qcom-next` | Stable variant identifier used in artifact and workspace identity. | +| `kernel-variant` | `qcom-next` | CI identifier used for artifact and workspace identity only. | +| `flavour` | `qcom-next` | Kernel flavour: the LOCALVERSION suffix, and so the kernel release identity. | | `suite` | `trixie` | Target suite. | | `ref-strategy` | `latest_tag` | `latest_tag`, `branch_tip`, or `pinned_ref`. | | `kernel-branch` | `qcom-next` | Branch for `branch_tip`, or immutable ref for `pinned_ref`; ignored by `latest_tag`. | @@ -497,9 +502,12 @@ To add a kernel variant: 1. Add one entry to `deliveries` per package the variant should produce: one per Daily suite and one per Release suite, each spelling out all of its own fields. Do not rely on another variant's values. -2. Keep `srcpkg`, `binpkg` and `kernel_config` identical across every entry for - the variant, and the ref fields identical across its entries of one `type`. - `resolve-matrix.py` rejects the matrix if they drift apart. +2. Give them all the same `flavour`, distinct from every other flavour's — it + becomes the LOCALVERSION suffix, so this is what lets the new kernel install + alongside the existing ones. Keep `srcpkg`, `binpkg` and `kernel_config` + identical across every entry for the flavour, and the ref fields identical + across its entries of one `type`. `resolve-matrix.py` rejects the matrix if + they drift apart. 3. Use `latest_tag` with a dated tag glob or `branch_tip` for Daily. Use `pinned_ref` for Release, and update that ref through a reviewed PR. 4. Give the variant distinct `srcpkg` and `binpkg` values. Set diff --git a/ci/build-matrix.yaml b/ci/build-matrix.yaml index 86df3c28..e356e359 100644 --- a/ci/build-matrix.yaml +++ b/ci/build-matrix.yaml @@ -12,11 +12,19 @@ # # Entries are written out in full rather than sharing YAML anchors, so each one # can be read, grepped, and changed on its own. resolve-matrix.py enforces the -# consistency that duplication would otherwise put at risk: a variant's entries +# consistency that duplication would otherwise put at risk: a flavour's entries # must agree on srcpkg, binpkg and kernel_config, its entries of one delivery # type must agree on the kernel ref they build, and its entries for one suite # must agree on dkms. # +# The two identifiers are not the same thing. flavour is the kernel's own +# identity: it becomes the LOCALVERSION suffix, so qcom-next and +# qcom-next-debug produce distinct kernel releases that install alongside each +# other, and every suite building one flavour produces the same kernel release. +# kernel_variant names the build in CI - the Actions job, the prepared-source +# artifact, the Debusine workspace, the S3 path - and never reaches a package +# name or version. +# # dkms varies by suite because it depends on which -dkms packages the # target archive carries: the Debian suites bundle camx and iris-vpu, which # Ubuntu does not package. @@ -34,6 +42,7 @@ deliveries: - kernel_variant: qcom-next type: Daily suite: trixie + flavour: qcom-next git_clone: https://github.com/qualcomm-linux/kernel branch_or_tag: qcom-next ref_strategy: latest_tag @@ -51,6 +60,7 @@ deliveries: - kernel_variant: qcom-next type: Daily suite: forky + flavour: qcom-next git_clone: https://github.com/qualcomm-linux/kernel branch_or_tag: qcom-next ref_strategy: latest_tag @@ -68,6 +78,7 @@ deliveries: - kernel_variant: qcom-next type: Daily suite: resolute + flavour: qcom-next git_clone: https://github.com/qualcomm-linux/kernel branch_or_tag: qcom-next ref_strategy: latest_tag @@ -83,6 +94,7 @@ deliveries: - kernel_variant: qcom-next type: Release suite: trixie + flavour: qcom-next git_clone: https://github.com/qualcomm-linux/kernel branch_or_tag: qcom-next-7.2-20260826 ref_strategy: pinned_ref @@ -100,6 +112,7 @@ deliveries: - kernel_variant: qcom-next type: Release suite: forky + flavour: qcom-next git_clone: https://github.com/qualcomm-linux/kernel branch_or_tag: qcom-next-7.2-20260826 ref_strategy: pinned_ref @@ -121,6 +134,7 @@ deliveries: - kernel_variant: qcom-next-debug type: Daily suite: trixie + flavour: qcom-next-debug git_clone: https://github.com/qualcomm-linux/kernel branch_or_tag: qcom-next ref_strategy: latest_tag @@ -140,6 +154,7 @@ deliveries: - kernel_variant: qcom-next-debug type: Daily suite: forky + flavour: qcom-next-debug git_clone: https://github.com/qualcomm-linux/kernel branch_or_tag: qcom-next ref_strategy: latest_tag @@ -159,6 +174,7 @@ deliveries: - kernel_variant: qcom-next-debug type: Release suite: trixie + flavour: qcom-next-debug git_clone: https://github.com/qualcomm-linux/kernel branch_or_tag: qcom-next-7.2-20260826 ref_strategy: pinned_ref @@ -178,6 +194,7 @@ deliveries: - kernel_variant: qcom-next-debug type: Release suite: forky + flavour: qcom-next-debug git_clone: https://github.com/qualcomm-linux/kernel branch_or_tag: qcom-next-7.2-20260826 ref_strategy: pinned_ref diff --git a/ci/scripts/derive-localversion.sh b/ci/scripts/derive-localversion.sh index eca3c989..0517e10e 100755 --- a/ci/scripts/derive-localversion.sh +++ b/ci/scripts/derive-localversion.sh @@ -3,23 +3,29 @@ # SPDX-License-Identifier: BSD-3-Clause-Clear set -euo pipefail -# Derive the version fields for a build from a kernel variant and resolved ref. +# Derive the version fields for a build from a kernel flavour and resolved ref. +# +# The flavour is the kernel's own identity, the part of the kernel release +# that distinguishes two kernels built from the same ref with different +# configuration, so that their linux-image packages install alongside each +# other. It is not the CI identifier for the build: a flavour is built for +# several suites, and all of those builds produce the same kernel release. # # Emits LOCALVERSION (the kernel release suffix), SNAPSHOT (the dated component # of the Debian version) and GITSHA, all derived from the ref in one place. # SNAPSHOT and GITSHA are emitted alongside rather than recovered from # LOCALVERSION later: reading them back out means guessing where each field ends -# in a string that also carries a variant name, and a hex SHA can end in eight +# in a string that also carries a flavour name, and a hex SHA can end in eight # digits of its own. # # For dated tag builds (ref ends in -YYYYMMDD, optionally .): -# Produces +-[.]-g<12 hex>. +# Produces +-[.]-g<12 hex>. # Example: qcom-next-7.2-rc3-20260722 -> +qcom-next-20260722-g07f50dc44edd # qcom-next-7.2-rc3-20260722.1 -> +qcom-next-20260722.1-g07f50dc44edd # # The respin ordinal distinguishes a second tag cut on the same day. It is # carried verbatim rather than normalised, so the first tag of a day stays -# plain +-: systemd compares the separator before the chunk +# plain +-: systemd compares the separator before the chunk # behind it, so an absent ordinal already sorts below a present one and no # build has to spell a ".0". # @@ -61,11 +67,11 @@ set -euo pipefail # which spells the release candidate ~rcN and orders correctly either way. # # Usage: -# ci/scripts/derive-localversion.sh --variant qcom-next --ref qcom-next-7.2-rc3-20260722 --sha 07f50dc44edd -# ci/scripts/derive-localversion.sh --variant arduino --ref main --sha 07f50dc44edd --date 20260904 +# ci/scripts/derive-localversion.sh --flavour qcom-next --ref qcom-next-7.2-rc3-20260722 --sha 07f50dc44edd +# ci/scripts/derive-localversion.sh --flavour arduino --ref main --sha 07f50dc44edd --date 20260904 # # Options: -# --variant VARIANT Kernel variant identifier. Defaults to qcom-next. +# --flavour FLAVOUR Kernel flavour. Defaults to qcom-next. # --ref REF Kernel ref (tag name or branch name). Required. # --sha SHA Commit SHA, truncated to 12 hex characters. Required. # --date DATE HEAD commit date as YYYYMMDD or YYYYMMDD.N. Required for @@ -86,7 +92,7 @@ set -euo pipefail # 0 Success. # 1 Error (invalid args, malformed --sha, branch tip without --date). -VARIANT="qcom-next" +FLAVOUR="qcom-next" REF="" SHA="" DATE="" @@ -98,7 +104,7 @@ usage() { while [[ $# -gt 0 ]]; do case $1 in - --variant) VARIANT="$2"; shift 2 ;; + --flavour) FLAVOUR="$2"; shift 2 ;; --ref) REF="$2"; shift 2 ;; --sha) SHA="$2"; shift 2 ;; --date) DATE="$2"; shift 2 ;; @@ -108,8 +114,8 @@ while [[ $# -gt 0 ]]; do done [[ -n "$REF" ]] || { echo "ERROR: --ref is required" >&2; exit 1; } -[[ "$VARIANT" =~ ^[a-z0-9]+([a-z0-9-]*[a-z0-9])?$ ]] || { - echo "ERROR: --variant must use lowercase letters, digits, and internal hyphens" >&2 +[[ "$FLAVOUR" =~ ^[a-z0-9]+([a-z0-9-]*[a-z0-9])?$ ]] || { + echo "ERROR: --flavour must use lowercase letters, digits, and internal hyphens" >&2 exit 1 } # Every build identifies its commit, so --sha is required for all of them, not @@ -123,11 +129,11 @@ done GITSHA="${SHA:0:12}" # Dated tags use a trailing YYYYMMDD snapshot, optionally followed by a respin -# ordinal. The matrix selects the tag set; the variant supplies the stable -# package identity used in LOCALVERSION. +# ordinal. The matrix selects the tag set; the flavour supplies the stable +# kernel identity used in LOCALVERSION. if [[ "$REF" =~ -([0-9]{8}(\.[0-9]+)?)$ ]]; then SNAPSHOT="${BASH_REMATCH[1]}" - LOCALVERSION="+${VARIANT}-${SNAPSHOT}-g${GITSHA}" + LOCALVERSION="+${FLAVOUR}-${SNAPSHOT}-g${GITSHA}" else # Branch-tip build: the ref carries no date, so the commit date supplies # one. Without it these builds had no snapshot at all, which put their @@ -141,7 +147,7 @@ else exit 1 } SNAPSHOT="$DATE" - LOCALVERSION="+${VARIANT}-${SNAPSHOT}-g${GITSHA}" + LOCALVERSION="+${FLAVOUR}-${SNAPSHOT}-g${GITSHA}" fi echo "LOCALVERSION=${LOCALVERSION}" diff --git a/ci/scripts/resolve-matrix.py b/ci/scripts/resolve-matrix.py index f92342cf..9c7eed52 100755 --- a/ci/scripts/resolve-matrix.py +++ b/ci/scripts/resolve-matrix.py @@ -77,6 +77,7 @@ "kernel_variant", "type", "suite", + "flavour", "git_clone", "branch_or_tag", "ref_strategy", @@ -97,19 +98,26 @@ REQUIRED_STRING_FIELDS + OPTIONAL_STRING_FIELDS + ("kernel_config", "dkms") ) -# Fields that identify the variant itself rather than one of its build legs. -# Every entry for a variant must agree on them, because they decide what the +# Everything about what is built is anchored on flavour, not kernel_variant. +# The flavour is the kernel's own identity: it becomes the LOCALVERSION +# suffix, so two flavours built from one ref produce distinct kernel releases +# whose linux-image packages install alongside each other. kernel_variant +# names the build in CI -- job names, artifacts, Debusine workspaces, S3 +# paths -- and carries no packaging meaning. + +# Fields that identify the flavour itself rather than one of its build legs. +# Every entry for a flavour must agree on them, because they decide what the # package is; the entries only differ in where it is delivered. -VARIANT_IDENTITY_FIELDS = ("srcpkg", "binpkg", "kernel_config") +FLAVOUR_IDENTITY_FIELDS = ("srcpkg", "binpkg", "kernel_config") -# Fields a variant may vary between suites but not within one. The out-of-tree +# Fields a flavour may vary between suites but not within one. The out-of-tree # module set depends on which -dkms packages the target archive has, so -# it is a property of the variant in a suite rather than of the variant. A +# it is a property of the flavour in a suite rather than of the flavour. A # suite's Daily and Release must still agree: a Daily that bundles a different # module set from the Release it precedes is not testing what will ship. SUITE_IDENTITY_FIELDS = ("dkms",) -# Fields deciding which kernel tree is built. All entries for one variant and +# Fields deciding which kernel tree is built. All entries for one flavour and # delivery type build the same source, so a stale suite cannot quietly ship a # different kernel from its siblings. REF_FIELDS = ("git_clone", "branch_or_tag", "ref_strategy", "tag_pattern") @@ -222,7 +230,7 @@ def check_entry(entry, report): for field in sorted(set(entry) - KNOWN_FIELDS): report(f"unknown field {field}") - for field in ("kernel_variant", "suite"): + for field in ("kernel_variant", "suite", "flavour"): value = entry.get(field) if isinstance(value, str) and not NAME_RE.match(value): report(f"{field} must use lowercase letters, digits, and internal hyphens") @@ -287,40 +295,47 @@ def describe(entry, index): def check_consistency(deliveries, errors): """Validate the invariants that span entries. - Entries are written out in full, so the matrix can state a variant twice + Entries are written out in full, so the matrix can state a flavour twice and disagree with itself. These checks are what makes that duplication safe to read at face value. + + Everything about the package is grouped by flavour. kernel_variant is + only checked for the one thing it is used for: naming a build leg + uniquely. """ by_leg = defaultdict(list) - by_variant = defaultdict(list) - by_variant_type = defaultdict(list) + by_flavour = defaultdict(list) + by_flavour_type = defaultdict(list) + by_flavour_suite = defaultdict(list) by_package_version = defaultdict(list) - variants_by_package = defaultdict(set) - by_variant_suite = defaultdict(list) + flavours_by_package = defaultdict(set) for entry in deliveries: if not isinstance(entry, dict): continue - variant = entry.get("kernel_variant") + flavour = entry.get("flavour") delivery_type = entry.get("type") suite = entry.get("suite") - if not isinstance(variant, str): + if not isinstance(flavour, str): continue - by_variant[variant].append(entry) - by_variant_type[(variant, delivery_type)].append(entry) - by_variant_suite[(variant, suite)].append(entry) - by_leg[(variant, delivery_type, suite)].append(entry) + by_flavour[flavour].append(entry) + by_flavour_type[(flavour, delivery_type)].append(entry) + by_flavour_suite[(flavour, suite)].append(entry) + by_leg[(entry.get("kernel_variant"), delivery_type, suite)].append(entry) for field in ("srcpkg", "binpkg"): if isinstance(entry.get(field), str) and entry[field]: - variants_by_package[(field, entry[field])].add(variant) + flavours_by_package[(field, entry[field])].add(flavour) if isinstance(entry.get("srcpkg"), str) and isinstance( entry.get("debian_revision"), str ): by_package_version[(entry["srcpkg"], entry["debian_revision"])].append(entry) + # kernel_variant plus suite is what names a build leg in the Actions UI and + # scopes its artifacts, workspace and S3 path, so a repeat would have two + # legs writing to one another's outputs. for (variant, delivery_type, suite), entries in sorted( by_leg.items(), key=lambda item: str(item[0]) ): @@ -330,50 +345,51 @@ def check_consistency(deliveries, errors): f"entries for suite {suite}; one entry is one generated package" ) - for variant, entries in sorted(by_variant.items()): - for field in VARIANT_IDENTITY_FIELDS: + for flavour, entries in sorted(by_flavour.items()): + for field in FLAVOUR_IDENTITY_FIELDS: values = {json.dumps(entry.get(field), sort_keys=True) for entry in entries} if len(values) > 1: errors.append( - f"kernel_variant {variant} must use one {field} across all its " + f"flavour {flavour} must use one {field} across all its " "entries (got " + ", ".join(sorted(values)) + ")" ) types = {entry.get("type") for entry in entries} for delivery_type in DELIVERY_TYPES: if delivery_type not in types: errors.append( - f"kernel_variant {variant} has no {delivery_type} entry; " - "every variant must define at least one of each" + f"flavour {flavour} has no {delivery_type} entry; " + "every flavour must define at least one of each" ) - for (variant, suite), entries in sorted( - by_variant_suite.items(), key=lambda item: str(item[0]) + for (flavour, suite), entries in sorted( + by_flavour_suite.items(), key=lambda item: str(item[0]) ): for field in SUITE_IDENTITY_FIELDS: values = {json.dumps(entry.get(field), sort_keys=True) for entry in entries} if len(values) > 1: errors.append( - f"kernel_variant {variant} must use one {field} across its " + f"flavour {flavour} must use one {field} across its " f"{suite} entries (got " + ", ".join(sorted(values)) + ")" ) - for (variant, delivery_type), entries in sorted( - by_variant_type.items(), key=lambda item: str(item[0]) + for (flavour, delivery_type), entries in sorted( + by_flavour_type.items(), key=lambda item: str(item[0]) ): for field in REF_FIELDS: values = {entry.get(field) for entry in entries} if len(values) > 1: rendered = ", ".join(sorted(str(v) for v in values)) errors.append( - f"kernel_variant {variant} must build one {field} across its " + f"flavour {flavour} must build one {field} across its " f"{delivery_type} entries (got {rendered})" ) - for (field, package), variants in sorted(variants_by_package.items()): - if len(variants) > 1: + # Two flavours sharing a package name would overwrite each other in the + # archive; the whole point of a second flavour is a second package. + for (field, package), flavours in sorted(flavours_by_package.items()): + if len(flavours) > 1: errors.append( - f"{field} {package} is shared by kernel variants " - + ", ".join(sorted(variants)) + f"{field} {package} is shared by flavours " + ", ".join(sorted(flavours)) ) for (srcpkg, revision), entries in sorted( @@ -388,7 +404,7 @@ def check_consistency(deliveries, errors): # A suite's Daily and Release differ only by the Daily's trailing ~, so the # Daily reliably sorts below the Release that supersedes it. - for variant, entries in sorted(by_variant.items()): + for flavour, entries in sorted(by_flavour.items()): revisions = { (entry.get("type"), entry.get("suite")): entry.get("debian_revision") for entry in entries @@ -401,7 +417,7 @@ def check_consistency(deliveries, errors): release = revisions.get(("Release", suite)) if isinstance(release, str) and revision != release + "~": errors.append( - f"kernel_variant {variant} suite {suite}: Daily " + f"flavour {flavour} suite {suite}: Daily " f'debian_revision "{revision}" must be the Release revision ' f'"{release}" with a trailing ~' ) From 478553cefefb5b647ced0c1cdbda61b740fb99ab Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 06/59] ci: name each build leg with its own kernel_variant kernel_variant repeated: three Daily entries all called qcom-next, distinguished only by their suite. The Actions job had to be named from two fields to say which leg it was, and a dispatch had to take a variant and a suite and offer scopes for each combination of the two. Give every entry a kernel_variant of its own - qcom-next-trixie, qcom-next-debug-forky - so a variant names exactly one build. The job name is now just the variant, a dispatch asks for builds by name, and resolve-matrix.py rejects a matrix that uses one variant twice in a delivery type. Uniqueness is per type rather than global: a run only ever resolves one type, so a Daily and the Release that supersedes it keep the same name, which is what makes them recognisable as the same leg. The dispatch filters take comma-separated lists, so a run can name several legs. --flavour joins them, because "release qcom-next everywhere it ships" is the normal release action and naming each suite individually would be a worse way to ask for it. A name matching no entry is now an error listing what is available, rather than a silently smaller build set - with per-leg names there is much more to mistype. Daily scopes become Full matrix, Selected variants, and Selected flavour; Release becomes Selected flavour and Selected variants. Renaming the legs would have moved every published artifact, since the S3 path, Debusine workspace and prepared-source artifact were all keyed on kernel_variant. They describe which package is at that location, not which CI job wrote it, so they are keyed on flavour now and their values are unchanged: qcom-next/trixie stays qcom-next/trixie. Verified leg by leg, along with LOCALVERSION. build-kernel-debusine.yml and build-kernel-ubuntu.yml take flavour in place of kernel-variant and no longer see the CI label at all. Signed-off-by: Christopher Obbard --- .github/workflows/build-kernel-deb.yml | 8 +- .github/workflows/build-kernel-debusine.yml | 10 +- .github/workflows/build-kernel-ubuntu.yml | 8 +- .github/workflows/daily.yml | 46 ++++---- .github/workflows/pr-build.yml | 2 +- .github/workflows/release.yml | 42 +++---- README.md | 51 +++++---- ci/build-matrix.yaml | 18 +-- ci/scripts/resolve-matrix.py | 119 +++++++++++++++----- 9 files changed, 189 insertions(+), 115 deletions(-) diff --git a/.github/workflows/build-kernel-deb.yml b/.github/workflows/build-kernel-deb.yml index f620d365..93c16fa3 100644 --- a/.github/workflows/build-kernel-deb.yml +++ b/.github/workflows/build-kernel-deb.yml @@ -6,7 +6,7 @@ on: kernel-variant: description: 'Kernel variant identifier, naming this build in CI only' required: true - default: 'qcom-next' + default: 'qcom-next-trixie' type: string flavour: description: 'Kernel flavour: the LOCALVERSION suffix and so the kernel release identity' @@ -421,7 +421,7 @@ jobs: - name: Upload prepared source tree artifact uses: actions/upload-artifact@v4 with: - name: kernel-srcpkg-${{ inputs.kernel-variant }}-${{ inputs.suite }} + name: kernel-srcpkg-${{ inputs.flavour }}-${{ inputs.suite }} path: /tmp/kernel-srcpkg.tar.gz if-no-files-found: error retention-days: 7 @@ -436,7 +436,7 @@ jobs: uses: ./.github/workflows/build-kernel-debusine.yml with: suite: ${{ inputs.suite || 'trixie' }} - kernel-variant: ${{ inputs.kernel-variant || 'qcom-next' }} + flavour: ${{ inputs.flavour || 'qcom-next' }} srcpkg: ${{ inputs.srcpkg || 'linux-qcom-next' }} debusine-parent-workspace: ${{ inputs.debusine-parent-workspace || vars.DEBUSINE_PARENT_WORKSPACE || 'qli-ci' }} target-workspace: ${{ inputs.target-workspace || '' }} @@ -455,4 +455,4 @@ jobs: uses: ./.github/workflows/build-kernel-ubuntu.yml with: distro: ${{ inputs.suite || 'resolute' }} - kernel-variant: ${{ inputs.kernel-variant || 'qcom-next' }} + flavour: ${{ inputs.flavour || 'qcom-next' }} diff --git a/.github/workflows/build-kernel-debusine.yml b/.github/workflows/build-kernel-debusine.yml index 7b8cc17a..6f733977 100644 --- a/.github/workflows/build-kernel-debusine.yml +++ b/.github/workflows/build-kernel-debusine.yml @@ -24,8 +24,8 @@ on: description: 'Target Debian suite (trixie, forky, sid)' type: string required: true - kernel-variant: - description: 'Kernel variant identifier used to isolate artifacts and Debusine workspaces' + flavour: + description: 'Kernel flavour, isolating artifacts, Debusine workspaces and S3 paths' type: string required: true srcpkg: @@ -90,7 +90,7 @@ jobs: - name: Download prepared source tree uses: actions/download-artifact@v4 with: - name: kernel-srcpkg-${{ inputs.kernel-variant }}-${{ inputs.suite }} + name: kernel-srcpkg-${{ inputs.flavour }}-${{ inputs.suite }} path: /tmp/srcpkg-artifact - name: Extract prepared source tree @@ -113,7 +113,7 @@ jobs: GITHUB_REPOSITORY_ID: ${{ github.repository_id }} GITHUB_RUN_ID: ${{ github.run_id }} GITHUB_RUN_ATTEMPT: ${{ github.run_attempt }} - JOB_INDEX: ${{ inputs.kernel-variant }}-${{ inputs.suite }} + JOB_INDEX: ${{ inputs.flavour }}-${{ inputs.suite }} DEBUSINE_HOST: ${{ vars.DEBUSINE_HOST }} DEBUSINE_SCOPE: ${{ vars.DEBUSINE_SCOPE }} DEBUSINE_USER: ${{ secrets.DEBUSINE_USER }} @@ -194,7 +194,7 @@ jobs: with: s3_bucket: ${{ vars.ARTIFACT_S3_BUCKET }} path: deb-artifacts - destination: ${{ github.repository_owner }}/pkg/debusine/${{ github.event.repository.name }}/${{ inputs.kernel-variant }}/${{ inputs.suite }}/${{ github.run_id }}-${{ github.run_attempt }}/ + destination: ${{ github.repository_owner }}/pkg/debusine/${{ github.event.repository.name }}/${{ inputs.flavour }}/${{ inputs.suite }}/${{ github.run_id }}-${{ github.run_attempt }}/ # --------------------------------------------------------------------------- # release: promote packages from CI workspace to target Debusine workspace. diff --git a/.github/workflows/build-kernel-ubuntu.yml b/.github/workflows/build-kernel-ubuntu.yml index f62317ec..97a15264 100644 --- a/.github/workflows/build-kernel-ubuntu.yml +++ b/.github/workflows/build-kernel-ubuntu.yml @@ -19,8 +19,8 @@ on: description: Target Ubuntu suite (noble, questing, resolute) type: string required: true - kernel-variant: - description: 'Kernel variant identifier used to isolate prepared-source artifacts and published packages' + flavour: + description: 'Kernel flavour, isolating prepared-source artifacts and published packages' type: string required: true @@ -60,7 +60,7 @@ jobs: - name: Download prepared source tree uses: actions/download-artifact@v4 with: - name: kernel-srcpkg-${{ inputs.kernel-variant }}-${{ inputs.distro }} + name: kernel-srcpkg-${{ inputs.flavour }}-${{ inputs.distro }} path: /tmp/srcpkg-artifact - name: Extract prepared source tree @@ -96,4 +96,4 @@ jobs: with: s3_bucket: ${{ vars.ARTIFACT_S3_BUCKET }} path: kernel-build/${{ inputs.distro }} - destination: ${{ github.repository_owner }}/pkg/temp/${{ github.event.repository.name }}/${{ inputs.kernel-variant }}/${{ inputs.distro }}/${{ github.run_id }}-${{ github.run_attempt }}/ + destination: ${{ github.repository_owner }}/pkg/temp/${{ github.event.repository.name }}/${{ inputs.flavour }}/${{ inputs.distro }}/${{ github.run_id }}-${{ github.run_attempt }}/ diff --git a/.github/workflows/daily.yml b/.github/workflows/daily.yml index e5d7cf4b..b5747857 100644 --- a/.github/workflows/daily.yml +++ b/.github/workflows/daily.yml @@ -2,9 +2,8 @@ name: daily # Daily kernel package builds. Runs at 3:00 PM PST (23:00 UTC), one parallel # build per Daily entry in ci/build-matrix.yaml. Manual dispatch can run the -# full Daily matrix, every suite for one kernel variant, or one kernel variant -# and suite. Variant and suite are matrix values, so future variants need no -# workflow edit. +# full Daily matrix, a named set of variants, or every suite of one flavour. +# Variants and flavours are matrix values, so new ones need no workflow edit. # # Matrix schema: ci/build-matrix.yaml # One entry is one generated package, so ci/scripts/resolve-matrix.py selects @@ -24,23 +23,23 @@ on: workflow_dispatch: inputs: build-scope: - description: 'Choose all variants, all suites for one variant, or one variant and suite' + description: 'Choose the whole matrix, named build(s), or every suite of one flavour' type: choice required: true - default: 'Selected variant and suite' + default: 'Selected variants' options: - 'Full matrix' - - 'Selected variant (all suites)' - - 'Selected variant and suite' - kernel-variant: - description: 'Kernel variant to build (ignored for Full matrix)' + - 'Selected variants' + - 'Selected flavour (all suites)' + kernel-variants: + description: 'Comma-separated variants to build, e.g. qcom-next-trixie,qcom-next-debug-forky (Selected variants)' required: true - default: 'qcom-next' + default: 'qcom-next-trixie' type: string - suite: - description: 'Target suite to build (used only for Selected variant and suite)' + flavour: + description: 'Kernel flavour to build in every suite it targets (Selected flavour)' required: true - default: 'trixie' + default: 'qcom-next' type: string permissions: @@ -53,7 +52,8 @@ jobs: # # Delegates to ci/scripts/resolve-matrix.py, which validates the whole matrix # and returns the entries with type: Daily. Manual dispatch selects the - # complete matrix, every suite for a variant, or one variant and suite. + # complete matrix, a comma-separated list of variants, or one flavour across + # every suite it targets. # --------------------------------------------------------------------------- configure-matrix: name: Configure build matrix @@ -67,8 +67,8 @@ jobs: id: set-matrix env: BUILD_SCOPE_INPUT: ${{ github.event.inputs.build-scope }} - INPUT_VARIANT: ${{ github.event.inputs.kernel-variant }} - INPUT_SUITE: ${{ github.event.inputs.suite }} + INPUT_VARIANTS: ${{ github.event.inputs.kernel-variants }} + INPUT_FLAVOUR: ${{ github.event.inputs.flavour }} run: | set -euo pipefail @@ -83,13 +83,13 @@ jobs: echo "Source: ci/build-matrix.yaml (full Daily matrix)" args=(--type Daily) ;; - "Selected variant (all suites)") - echo "Source: ci/build-matrix.yaml (variant: $INPUT_VARIANT, all suites)" - args=(--type Daily --kernel-variant "$INPUT_VARIANT") + "Selected variants") + echo "Source: ci/build-matrix.yaml (variants: $INPUT_VARIANTS)" + args=(--type Daily --kernel-variant "$INPUT_VARIANTS") ;; - "Selected variant and suite") - echo "Source: ci/build-matrix.yaml (variant: $INPUT_VARIANT, suite: $INPUT_SUITE)" - args=(--type Daily --kernel-variant "$INPUT_VARIANT" --suite "$INPUT_SUITE") + "Selected flavour (all suites)") + echo "Source: ci/build-matrix.yaml (flavour: $INPUT_FLAVOUR, all suites)" + args=(--type Daily --flavour "$INPUT_FLAVOUR") ;; *) echo "ERROR: unsupported build scope: $BUILD_SCOPE" >&2 @@ -110,7 +110,7 @@ jobs: # build does not cancel the forky build. # --------------------------------------------------------------------------- build: - name: ${{ matrix.kernel_variant }} (${{ matrix.suite }}) + name: ${{ matrix.kernel_variant }} needs: configure-matrix strategy: fail-fast: false diff --git a/.github/workflows/pr-build.yml b/.github/workflows/pr-build.yml index 4ef4263a..65943df8 100644 --- a/.github/workflows/pr-build.yml +++ b/.github/workflows/pr-build.yml @@ -52,7 +52,7 @@ jobs: # resolute results on the PR. # --------------------------------------------------------------------------- build: - name: ${{ matrix.kernel_variant }} (${{ matrix.suite }}) + name: ${{ matrix.kernel_variant }} needs: configure-matrix strategy: fail-fast: false diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 80707bc6..6df0c322 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -8,9 +8,9 @@ name: release # target workspace (e.g. qli) via debusine-action lib/release. # # To update a release ref: open a PR that changes branch_or_tag on every -# Release entry for that kernel variant in ci/build-matrix.yaml. A variant's -# Release entries must agree on the ref, so resolve-matrix.py rejects the -# matrix if only some of them are updated. +# Release entry for that flavour in ci/build-matrix.yaml. A flavour's Release +# entries must agree on the ref, so resolve-matrix.py rejects the matrix if +# only some of them are updated. # # This workflow intentionally has no schedule trigger. Releases are always # initiated manually after validation. @@ -19,22 +19,22 @@ on: workflow_dispatch: inputs: release-scope: - description: 'Choose all suites for one variant or one variant and suite' + description: 'Choose every suite of one flavour, or named build(s)' type: choice required: true - default: 'Selected variant (all suites)' + default: 'Selected flavour (all suites)' options: - - 'Selected variant (all suites)' - - 'Selected variant and suite' - kernel-variant: - description: 'Kernel variant to release' + - 'Selected flavour (all suites)' + - 'Selected variants' + flavour: + description: 'Kernel flavour to release in every suite it targets (Selected flavour)' required: true default: 'qcom-next' type: string - suite: - description: 'Target suite to release (used only for Selected variant and suite)' + kernel-variants: + description: 'Comma-separated variants to release, e.g. qcom-next-trixie,qcom-next-forky (Selected variants)' required: true - default: 'trixie' + default: 'qcom-next-trixie' type: string permissions: @@ -59,21 +59,21 @@ jobs: id: set-matrix env: RELEASE_SCOPE_INPUT: ${{ github.event.inputs.release-scope }} - INPUT_VARIANT: ${{ github.event.inputs.kernel-variant }} - INPUT_SUITE: ${{ github.event.inputs.suite }} + INPUT_FLAVOUR: ${{ github.event.inputs.flavour }} + INPUT_VARIANTS: ${{ github.event.inputs.kernel-variants }} run: | set -euo pipefail RELEASE_SCOPE="$RELEASE_SCOPE_INPUT" case "$RELEASE_SCOPE" in - "Selected variant (all suites)") - echo "Source: ci/build-matrix.yaml (variant: $INPUT_VARIANT, all suites)" - args=(--type Release --kernel-variant "$INPUT_VARIANT") + "Selected flavour (all suites)") + echo "Source: ci/build-matrix.yaml (flavour: $INPUT_FLAVOUR, all suites)" + args=(--type Release --flavour "$INPUT_FLAVOUR") ;; - "Selected variant and suite") - echo "Source: ci/build-matrix.yaml (variant: $INPUT_VARIANT, suite: $INPUT_SUITE)" - args=(--type Release --kernel-variant "$INPUT_VARIANT" --suite "$INPUT_SUITE") + "Selected variants") + echo "Source: ci/build-matrix.yaml (variants: $INPUT_VARIANTS)" + args=(--type Release --kernel-variant "$INPUT_VARIANTS") ;; *) echo "ERROR: unsupported release scope: $RELEASE_SCOPE" >&2 @@ -93,7 +93,7 @@ jobs: # release path (lib/release) instead of the S3 publish path. # --------------------------------------------------------------------------- build: - name: ${{ matrix.kernel_variant }} (${{ matrix.suite }}) + name: ${{ matrix.kernel_variant }} needs: configure-matrix strategy: fail-fast: false diff --git a/README.md b/README.md index 4c7b452a..ca9ab023 100644 --- a/README.md +++ b/README.md @@ -43,12 +43,14 @@ Two entry points use the same reusable build pipeline: to the selected production Debusine workspace. One entry in `deliveries` is one generated package: a single `kernel_variant`, -a single `type`, and a single `suite`. Nothing is expanded or derived at -resolve time, so what an entry says is what gets built: +a single `type`, and a single `suite`. The `kernel_variant` names that one +build and nothing else — it is the Actions job name and what a manual dispatch +asks for. Nothing is expanded or derived at resolve time, so what an entry says +is what gets built: ```yaml deliveries: - - kernel_variant: qcom-next + - kernel_variant: qcom-next-trixie type: Daily suite: trixie flavour: qcom-next @@ -66,7 +68,7 @@ deliveries: - audioreach debian_revision: '0qli1~bpo13+1~' - - kernel_variant: qcom-next + - kernel_variant: qcom-next-trixie type: Release suite: trixie flavour: qcom-next @@ -132,8 +134,10 @@ Daily is the recurring build and artifact-publication path. - The scheduled run resolves the full `Daily` matrix. - A manual run selects one **Build scope**: - **Full matrix** builds every configured variant and suite. - - **Selected variant (all suites)** builds every configured suite for one variant. - - **Selected variant and suite** builds one isolated matrix leg. + - **Selected variants** builds a comma-separated list of `kernel_variant` + names, e.g. `qcom-next-trixie,qcom-next-debug-forky`. + - **Selected flavour (all suites)** builds one flavour in every suite it + targets. - `latest_tag` resolves the newest matching dated tag; `branch_tip` resolves the configured branch directly. - Debian suites build in Debusine, then their `.deb` outputs are downloaded and @@ -146,13 +150,13 @@ Daily is the recurring build and artifact-publication path. Release is the controlled promotion path. - It is manual only and uses one **Release scope** for a kernel variant: - - **Selected variant (all suites)** is the normal release action and promotes - every configured Release suite for that variant. - - **Selected variant and suite** promotes one configured Release suite for - that variant when a targeted action is required. + - **Selected flavour (all suites)** is the normal release action and promotes + every configured Release suite for that flavour. + - **Selected variants** promotes a comma-separated list of `kernel_variant` + names when a targeted action is required. - It uses the pinned `branch_or_tag` from the selected `Release` matrix row; it never resolves a newest tag. -- Debian source and binary artifacts are built in per-variant, per-suite +- Debian source and binary artifacts are built in per-flavour, per-suite Debusine CI workspaces. - Successful builds are promoted with Debusine's `package-publish` workflow to the `qli` workspace, where they are available through the production Debusine @@ -175,7 +179,7 @@ them to the workflow matrix as they stand. Each entry carries: | Field | Purpose | | --- | --- | -| `kernel_variant` | CI identifier for this build: its Actions job name, prepared-source artifact, Debusine child workspace and S3 path. Never reaches a package name or version. Lowercase letters, digits, and internal hyphens only. | +| `kernel_variant` | The name of this one build, and nothing else: its Actions job name, and what a manual dispatch asks for. Unique within a delivery type, so a Daily and its Release may share a name. Never reaches a package name, a version, or a published path. Lowercase letters, digits, and internal hyphens only. | | `type` | `Daily` or `Release`. | | `suite` | The one suite this entry builds for. | | `flavour` | The kernel's own identity, and the only matrix field that reaches the built kernel. `derive-localversion.sh` makes it the LOCALVERSION suffix, so two flavours built from one ref get distinct kernel releases and their `linux-image` packages coexist. All entries sharing a flavour build the same package for different suites. | @@ -206,8 +210,9 @@ Because entries are written out in full, the resolver also checks the invariants that span them, which is what makes the duplication safe to read at face value: -- No two entries share a `kernel_variant`, `type` and `suite` — one entry is - one generated package. +- No two entries of one `type` share a `kernel_variant` — a variant names + exactly one build, so a run cannot produce two jobs with one name and a + dispatch cannot be ambiguous. - A flavour's entries agree on `srcpkg`, `binpkg` and `kernel_config`; those decide what the package *is*, and the entries differ only in where it goes. - A flavour's entries for one suite agree on `dkms`. The module set depends on @@ -229,20 +234,21 @@ the variant and suite it was given (`resolve-matrix.py --field debian_revision`) and builds at the revision the daily build would have used. Each entry has a distinct prepared-source artifact, Debusine child workspace, -and S3 path keyed by `kernel_variant + suite`. This prevents two variants that -both build, for example, `trixie` from consuming or publishing each other's -inputs or outputs. +and S3 path keyed by `flavour + suite`. This prevents two flavours that both +build, for example, `trixie` from consuming or publishing each other's inputs +or outputs. They key on `flavour`, not `kernel_variant`, so renaming a build +leg never moves a published artifact. Daily S3 outputs use these layouts, where `` is `-`: ```text -/pkg/debusine///// -/pkg/temp///// +/pkg/debusine///// +/pkg/temp///// ``` The first layout is for Debian/Debusine builds; the second is for Ubuntu Docker -builds. Consumers must select the intended kernel variant and suite. +builds. Consumers must select the intended flavour and suite. Supporting scripts keep workflow YAML small and testable: @@ -452,7 +458,7 @@ The available inputs are: | Input | Default | Purpose | | --- | --- | --- | -| `kernel-variant` | `qcom-next` | CI identifier used for artifact and workspace identity only. | +| `kernel-variant` | `qcom-next-trixie` | Names this build; also selects the matrix entry a blank `debian-revision` is taken from. | | `flavour` | `qcom-next` | Kernel flavour: the LOCALVERSION suffix, and so the kernel release identity. | | `suite` | `trixie` | Target suite. | | `ref-strategy` | `latest_tag` | `latest_tag`, `branch_tip`, or `pinned_ref`. | @@ -501,7 +507,8 @@ To add a kernel variant: 1. Add one entry to `deliveries` per package the variant should produce: one per Daily suite and one per Release suite, each spelling out all of its own - fields. Do not rely on another variant's values. + fields, and each with a `kernel_variant` unique within its delivery type. + Do not rely on another variant's values. 2. Give them all the same `flavour`, distinct from every other flavour's — it becomes the LOCALVERSION suffix, so this is what lets the new kernel install alongside the existing ones. Keep `srcpkg`, `binpkg` and `kernel_config` diff --git a/ci/build-matrix.yaml b/ci/build-matrix.yaml index e356e359..3d5bd768 100644 --- a/ci/build-matrix.yaml +++ b/ci/build-matrix.yaml @@ -39,7 +39,7 @@ deliveries: # --------------------------------------------------------------------------- # qcom-next — the standard kernel. # --------------------------------------------------------------------------- - - kernel_variant: qcom-next + - kernel_variant: qcom-next-trixie type: Daily suite: trixie flavour: qcom-next @@ -57,7 +57,7 @@ deliveries: - audioreach debian_revision: '0qli1~bpo13+1~' - - kernel_variant: qcom-next + - kernel_variant: qcom-next-forky type: Daily suite: forky flavour: qcom-next @@ -75,7 +75,7 @@ deliveries: - audioreach debian_revision: '0qli1~' - - kernel_variant: qcom-next + - kernel_variant: qcom-next-resolute type: Daily suite: resolute flavour: qcom-next @@ -91,7 +91,7 @@ deliveries: - kgsl debian_revision: '0qli1~26.04.1~' - - kernel_variant: qcom-next + - kernel_variant: qcom-next-trixie type: Release suite: trixie flavour: qcom-next @@ -109,7 +109,7 @@ deliveries: debian_revision: '0qli1~bpo13+1' target_workspace: qli - - kernel_variant: qcom-next + - kernel_variant: qcom-next-forky type: Release suite: forky flavour: qcom-next @@ -131,7 +131,7 @@ deliveries: # qcom-next-debug — the same kernel ref built with the in-tree debug # fragments, packaged separately so it installs alongside qcom-next. # --------------------------------------------------------------------------- - - kernel_variant: qcom-next-debug + - kernel_variant: qcom-next-debug-trixie type: Daily suite: trixie flavour: qcom-next-debug @@ -151,7 +151,7 @@ deliveries: - audioreach debian_revision: '0qli1~bpo13+1~' - - kernel_variant: qcom-next-debug + - kernel_variant: qcom-next-debug-forky type: Daily suite: forky flavour: qcom-next-debug @@ -171,7 +171,7 @@ deliveries: - audioreach debian_revision: '0qli1~' - - kernel_variant: qcom-next-debug + - kernel_variant: qcom-next-debug-trixie type: Release suite: trixie flavour: qcom-next-debug @@ -191,7 +191,7 @@ deliveries: debian_revision: '0qli1~bpo13+1' target_workspace: qli - - kernel_variant: qcom-next-debug + - kernel_variant: qcom-next-debug-forky type: Release suite: forky flavour: qcom-next-debug diff --git a/ci/scripts/resolve-matrix.py b/ci/scripts/resolve-matrix.py index 9c7eed52..abe80be2 100755 --- a/ci/scripts/resolve-matrix.py +++ b/ci/scripts/resolve-matrix.py @@ -15,22 +15,32 @@ Usage: ci/scripts/resolve-matrix.py --type Daily - ci/scripts/resolve-matrix.py --type Release --kernel-variant qcom-next - ci/scripts/resolve-matrix.py --type Daily --kernel-variant qcom-next --suite trixie - ci/scripts/resolve-matrix.py --type Daily --kernel-variant qcom-next \\ - --suite trixie --field debian_revision + ci/scripts/resolve-matrix.py --type Daily --kernel-variant qcom-next-trixie + ci/scripts/resolve-matrix.py --type Daily --kernel-variant qcom-next-trixie,qcom-next-forky + ci/scripts/resolve-matrix.py --type Release --flavour qcom-next + ci/scripts/resolve-matrix.py --type Daily --kernel-variant qcom-next-trixie \\ + --field debian_revision Options: --type TYPE Delivery type to select (Daily or Release). Required. - --kernel-variant VARIANT Select only this kernel variant. - --suite SUITE Select only this suite. + --kernel-variant VARIANTS Select only these kernel variants, comma-separated. + A variant names one leg, so this is the way to + ask for a specific set of builds. + --flavour FLAVOURS Select only these flavours, comma-separated. A + flavour spans its suites, so this asks for one + kernel everywhere it is built. + --suite SUITES Select only these suites, comma-separated. --field NAME Print just this field of the single selected entry, unquoted. Errors unless exactly one entry matches. --matrix-file FILE Matrix path (default: ci/build-matrix.yaml relative to CWD). + Filters combine: an entry must match every filter given. Every name in a + filter must match at least one entry of the selected type, so a typo or a + stale variant fails instead of quietly narrowing the build set. + Output: Without --field, a compact JSON array of the selected entries, ready for a GitHub Actions matrix `include`. kernel_config and dkms are joined into the @@ -322,7 +332,7 @@ def check_consistency(deliveries, errors): by_flavour[flavour].append(entry) by_flavour_type[(flavour, delivery_type)].append(entry) by_flavour_suite[(flavour, suite)].append(entry) - by_leg[(entry.get("kernel_variant"), delivery_type, suite)].append(entry) + by_leg[(delivery_type, entry.get("kernel_variant"))].append(entry) for field in ("srcpkg", "binpkg"): if isinstance(entry.get(field), str) and entry[field]: @@ -333,16 +343,19 @@ def check_consistency(deliveries, errors): ): by_package_version[(entry["srcpkg"], entry["debian_revision"])].append(entry) - # kernel_variant plus suite is what names a build leg in the Actions UI and - # scopes its artifacts, workspace and S3 path, so a repeat would have two - # legs writing to one another's outputs. - for (variant, delivery_type, suite), entries in sorted( + # kernel_variant names one build leg: it is the Actions job name and what a + # workflow dispatch asks for by name. It only has to be unique within a + # delivery type, because a run only ever resolves one type, and that lets a + # Daily and its Release share a name. Two entries sharing both would give a + # run two identically named jobs and make the dispatch filter ambiguous. + for (delivery_type, variant), entries in sorted( by_leg.items(), key=lambda item: str(item[0]) ): if len(entries) > 1: + suites = ", ".join(sorted(str(e.get("suite")) for e in entries)) errors.append( - f"kernel_variant {variant} defines {len(entries)} {delivery_type} " - f"entries for suite {suite}; one entry is one generated package" + f"kernel_variant {variant} is used by {len(entries)} {delivery_type} " + f"entries (suites {suites}); a variant names exactly one build" ) for flavour, entries in sorted(by_flavour.items()): @@ -477,24 +490,56 @@ def load_matrix(path): return deliveries -def select(deliveries, delivery_type, kernel_variant, suite): - """Return the entries matching the requested filters, in matrix order.""" +def parse_filter(value): + """Split a comma-separated filter into names, or None when unset.""" + if not value: + return None + return [name.strip() for name in value.split(",") if name.strip()] + + +def select(deliveries, delivery_type, filters): + """Return the entries matching the requested filters, in matrix order. + + filters maps a field name to the list of values allowed for it, or to + None when that filter was not given. + """ return [ entry for entry in deliveries if entry["type"] == delivery_type - and kernel_variant in ("", entry["kernel_variant"]) - and suite in ("", entry["suite"]) + and all( + wanted is None or entry[field] in wanted + for field, wanted in filters.items() + ) ] -def describe_selection(delivery_type, kernel_variant, suite): +def unmatched_filters(deliveries, delivery_type, filters): + """Names asked for that no entry of this delivery type offers.""" + missing = [] + for field, wanted in filters.items(): + if wanted is None: + continue + available = { + entry[field] for entry in deliveries if entry["type"] == delivery_type + } + for name in wanted: + if name not in available: + missing.append( + f"no {delivery_type} entry has {field} {name} " + f"(available: {', '.join(sorted(available))})" + ) + return missing + + +def describe_selection(delivery_type, filters): """Render the active filters for an error message.""" parts = [f"type={delivery_type}"] - if kernel_variant: - parts.append(f"kernel_variant={kernel_variant}") - if suite: - parts.append(f"suite={suite}") + parts += [ + f"{field}={','.join(wanted)}" + for field, wanted in filters.items() + if wanted is not None + ] return " ".join(parts) @@ -516,9 +561,16 @@ def main(): "--type", required=True, choices=DELIVERY_TYPES, help="delivery type to select" ) parser.add_argument( - "--kernel-variant", default="", help="select only this kernel variant" + "--kernel-variant", + default="", + help="select only these kernel variants, comma-separated", + ) + parser.add_argument( + "--flavour", default="", help="select only these flavours, comma-separated" + ) + parser.add_argument( + "--suite", default="", help="select only these suites, comma-separated" ) - parser.add_argument("--suite", default="", help="select only this suite") parser.add_argument( "--field", default="", @@ -530,9 +582,24 @@ def main(): args = parser.parse_args() deliveries = load_matrix(args.matrix_file) - selected = select(deliveries, args.type, args.kernel_variant, args.suite) + filters = { + "kernel_variant": parse_filter(args.kernel_variant), + "flavour": parse_filter(args.flavour), + "suite": parse_filter(args.suite), + } + what = describe_selection(args.type, filters) + + # Report a name that matches nothing before reporting an empty selection: + # "no Daily entry has kernel_variant qcom-next" says what to fix, where + # "no entries found" leaves the reader to work out which filter was wrong. + unmatched = unmatched_filters(deliveries, args.type, filters) + if unmatched: + sys.exit( + f"ERROR: Nothing to build for {what}:\n" + + "\n".join(f" - {problem}" for problem in unmatched) + ) - what = describe_selection(args.type, args.kernel_variant, args.suite) + selected = select(deliveries, args.type, filters) if not selected: sys.exit(f"ERROR: No matrix entries found for {what}") From c4fafb5c9b56e5f7179b621513c1d085bcb14097 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 07/59] ci: rename the matrix keys to builds and name The two keys were named for what the matrix used to be. "deliveries" described rows that were expanded into build legs, and "kernel_variant" identified a kernel variant that several rows shared. Neither is true any more: an entry is one build, and its identifier labels that one build and nothing else, having been separated from flavour and from every published path. Rename them to builds and name, which is what a YAML list of things each carrying an identifier normally looks like, and reads as such: builds: - name: qcom-next-trixie type: Daily suite: trixie flavour: qcom-next The resolver's filter follows: --kernel-variant becomes --build, and build-kernel-deb.yml's kernel-variant input becomes build. The dispatch inputs on daily.yml and release.yml become "builds", taking the same comma-separated list, and their scopes read "Selected builds". Both old spellings now fail rather than being ignored: deliveries is rejected as an unknown top-level key, and kernel_variant as an unknown field on an entry whose name is missing. Names only. No build changes what it produces: the job names, S3 paths, Debusine workspaces, revisions and LOCALVERSIONs are all as they were. Signed-off-by: Christopher Obbard --- .github/workflows/build-kernel-deb.yml | 30 ++--- .github/workflows/build-kernel-debusine.yml | 2 +- .github/workflows/daily.yml | 32 +++--- .github/workflows/pr-build.yml | 4 +- .github/workflows/release.yml | 16 +-- README.md | 55 +++++----- ci/build-matrix.yaml | 31 +++--- ci/scripts/resolve-matrix.py | 116 ++++++++++---------- 8 files changed, 143 insertions(+), 143 deletions(-) diff --git a/.github/workflows/build-kernel-deb.yml b/.github/workflows/build-kernel-deb.yml index 93c16fa3..07f6a668 100644 --- a/.github/workflows/build-kernel-deb.yml +++ b/.github/workflows/build-kernel-deb.yml @@ -3,8 +3,8 @@ name: build-kernel-deb on: workflow_dispatch: inputs: - kernel-variant: - description: 'Kernel variant identifier, naming this build in CI only' + build: + description: 'Name of this build, as listed in ci/build-matrix.yaml' required: true default: 'qcom-next-trixie' type: string @@ -86,10 +86,10 @@ on: # Called by daily.yml and release.yml as a reusable workflow. workflow_call: inputs: - kernel-variant: - description: 'Kernel variant identifier, naming this build in CI only' + build: + description: 'Name of this build, as listed in ci/build-matrix.yaml' type: string - default: 'qcom-next' + default: 'qcom-next-trixie' flavour: description: 'Kernel flavour: the LOCALVERSION suffix and so the kernel release identity' type: string @@ -180,12 +180,12 @@ jobs: shell: bash env: SUITE_INPUT: ${{ inputs.suite || 'trixie' }} - KERNEL_VARIANT_INPUT: ${{ inputs.kernel-variant || 'qcom-next' }} + BUILD_INPUT: ${{ inputs.build || 'qcom-next-trixie' }} run: | suite="$SUITE_INPUT" - kernel_variant="$KERNEL_VARIANT_INPUT" - [[ "$kernel_variant" =~ ^[a-z0-9]+([a-z0-9-]*[a-z0-9])?$ ]] || { - echo "ERROR: kernel variant must use lowercase letters, digits, and internal hyphens" >&2 + build_name="$BUILD_INPUT" + [[ "$build_name" =~ ^[a-z0-9]+([a-z0-9-]*[a-z0-9])?$ ]] || { + echo "ERROR: build name must use lowercase letters, digits, and internal hyphens" >&2 exit 1 } case "$suite" in @@ -246,7 +246,7 @@ jobs: BRANCH_INPUT: ${{ inputs.kernel-branch || 'qcom-next' }} REF_STRATEGY_INPUT: ${{ inputs.ref-strategy || 'latest_tag' }} TAG_PATTERN_INPUT: ${{ inputs.tag-pattern || 'qcom-next-*' }} - KERNEL_VARIANT_INPUT: ${{ inputs.kernel-variant || 'qcom-next' }} + BUILD_INPUT: ${{ inputs.build || 'qcom-next-trixie' }} run: | set -euo pipefail BRANCH="$BRANCH_INPUT" @@ -269,7 +269,7 @@ jobs: esac echo "KERNEL_REF=$KERNEL_REF" >> "$GITHUB_ENV" - echo "Kernel variant: $KERNEL_VARIANT_INPUT" + echo "Build: $BUILD_INPUT" echo "Ref strategy: $REF_STRATEGY" echo "Resolved kernel ref: $KERNEL_REF" @@ -312,7 +312,7 @@ jobs: # direct dispatch is build-only and never promotes. env: DEBIAN_REVISION_INPUT: ${{ inputs.debian-revision }} - KERNEL_VARIANT_INPUT: ${{ inputs.kernel-variant || 'qcom-next' }} + BUILD_INPUT: ${{ inputs.build || 'qcom-next-trixie' }} run: | set -euo pipefail INPUT_REVISION="$DEBIAN_REVISION_INPUT" @@ -322,14 +322,14 @@ jobs: else DEBIAN_REVISION=$(ci/scripts/resolve-matrix.py \ --type Daily \ - --kernel-variant "$KERNEL_VARIANT_INPUT" \ + --build "$BUILD_INPUT" \ --suite "$DISTRO" \ --field debian_revision) || { - echo "::error::No Daily matrix entry for kernel-variant=$KERNEL_VARIANT_INPUT suite=$DISTRO." + echo "::error::No Daily matrix entry named $BUILD_INPUT for suite $DISTRO." echo "::error::Add one to ci/build-matrix.yaml, or set the debian-revision input to build this combination ad hoc." exit 1 } - echo "Matrix Debian revision for $KERNEL_VARIANT_INPUT/$DISTRO: $DEBIAN_REVISION" + echo "Matrix Debian revision for $BUILD_INPUT/$DISTRO: $DEBIAN_REVISION" fi echo "DEBIAN_REVISION=$DEBIAN_REVISION" >> "$GITHUB_ENV" diff --git a/.github/workflows/build-kernel-debusine.yml b/.github/workflows/build-kernel-debusine.yml index 6f733977..2f21529b 100644 --- a/.github/workflows/build-kernel-debusine.yml +++ b/.github/workflows/build-kernel-debusine.yml @@ -5,7 +5,7 @@ name: build-debusine # Called by build-kernel-deb.yml for Debian-family suites (trixie, forky, sid). # The kernel source is already prepared by the caller's prepare job and shared # as the kernel-srcpkg artifact. Artifact and workspace identity include both -# kernel_variant and suite so parallel variants cannot share inputs or outputs. +# flavour and suite so parallel flavours cannot share inputs or outputs. # # Publish path is determined by target-workspace: # target-workspace == '' -> daily path: download .deb files, upload to S3. diff --git a/.github/workflows/daily.yml b/.github/workflows/daily.yml index b5747857..06abc389 100644 --- a/.github/workflows/daily.yml +++ b/.github/workflows/daily.yml @@ -2,16 +2,16 @@ name: daily # Daily kernel package builds. Runs at 3:00 PM PST (23:00 UTC), one parallel # build per Daily entry in ci/build-matrix.yaml. Manual dispatch can run the -# full Daily matrix, a named set of variants, or every suite of one flavour. -# Variants and flavours are matrix values, so new ones need no workflow edit. +# full Daily matrix, a named set of builds, or every suite of one flavour. +# Build names and flavours are matrix values, so new ones need no workflow edit. # # Matrix schema: ci/build-matrix.yaml # One entry is one generated package, so ci/scripts/resolve-matrix.py selects # the entries with type: Daily and passes them through as they stand. Every -# field (kernel_variant, suite, flavour, srcpkg, binpkg, kernel_config, dkms, +# field (name, suite, flavour, srcpkg, binpkg, kernel_config, dkms, # debian_revision, git_clone, branch_or_tag, ref_strategy) becomes a -# build-kernel-deb.yml input. kernel_variant names the job and scopes its -# artifacts; flavour is what reaches the built kernel's version. +# build-kernel-deb.yml input. name labels the job; flavour is what reaches +# the built kernel's version and its published paths. # # To add a daily build target: add an entry to ci/build-matrix.yaml with # type: Daily. No workflow changes needed. @@ -26,13 +26,13 @@ on: description: 'Choose the whole matrix, named build(s), or every suite of one flavour' type: choice required: true - default: 'Selected variants' + default: 'Selected builds' options: - 'Full matrix' - - 'Selected variants' + - 'Selected builds' - 'Selected flavour (all suites)' - kernel-variants: - description: 'Comma-separated variants to build, e.g. qcom-next-trixie,qcom-next-debug-forky (Selected variants)' + builds: + description: 'Comma-separated build names, e.g. qcom-next-trixie,qcom-next-debug-forky (Selected builds)' required: true default: 'qcom-next-trixie' type: string @@ -52,8 +52,8 @@ jobs: # # Delegates to ci/scripts/resolve-matrix.py, which validates the whole matrix # and returns the entries with type: Daily. Manual dispatch selects the - # complete matrix, a comma-separated list of variants, or one flavour across - # every suite it targets. + # complete matrix, a comma-separated list of build names, or one flavour + # across every suite it targets. # --------------------------------------------------------------------------- configure-matrix: name: Configure build matrix @@ -67,7 +67,7 @@ jobs: id: set-matrix env: BUILD_SCOPE_INPUT: ${{ github.event.inputs.build-scope }} - INPUT_VARIANTS: ${{ github.event.inputs.kernel-variants }} + INPUT_BUILDS: ${{ github.event.inputs.builds }} INPUT_FLAVOUR: ${{ github.event.inputs.flavour }} run: | set -euo pipefail @@ -84,8 +84,8 @@ jobs: args=(--type Daily) ;; "Selected variants") - echo "Source: ci/build-matrix.yaml (variants: $INPUT_VARIANTS)" - args=(--type Daily --kernel-variant "$INPUT_VARIANTS") + echo "Source: ci/build-matrix.yaml (builds: $INPUT_BUILDS)" + args=(--type Daily --build "$INPUT_BUILDS") ;; "Selected flavour (all suites)") echo "Source: ci/build-matrix.yaml (flavour: $INPUT_FLAVOUR, all suites)" @@ -110,7 +110,7 @@ jobs: # build does not cancel the forky build. # --------------------------------------------------------------------------- build: - name: ${{ matrix.kernel_variant }} + name: ${{ matrix.name }} needs: configure-matrix strategy: fail-fast: false @@ -119,7 +119,7 @@ jobs: uses: ./.github/workflows/build-kernel-deb.yml with: suite: ${{ matrix.suite }} - kernel-variant: ${{ matrix.kernel_variant }} + build: ${{ matrix.name }} flavour: ${{ matrix.flavour }} git-clone: ${{ matrix.git_clone }} kernel-branch: ${{ matrix.branch_or_tag }} diff --git a/.github/workflows/pr-build.yml b/.github/workflows/pr-build.yml index 65943df8..df03c318 100644 --- a/.github/workflows/pr-build.yml +++ b/.github/workflows/pr-build.yml @@ -52,7 +52,7 @@ jobs: # resolute results on the PR. # --------------------------------------------------------------------------- build: - name: ${{ matrix.kernel_variant }} + name: ${{ matrix.name }} needs: configure-matrix strategy: fail-fast: false @@ -61,7 +61,7 @@ jobs: uses: ./.github/workflows/build-kernel-deb.yml with: suite: ${{ matrix.suite }} - kernel-variant: ${{ matrix.kernel_variant }} + build: ${{ matrix.name }} flavour: ${{ matrix.flavour }} git-clone: ${{ matrix.git_clone }} kernel-branch: ${{ matrix.branch_or_tag }} diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 6df0c322..e2e55206 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -25,14 +25,14 @@ on: default: 'Selected flavour (all suites)' options: - 'Selected flavour (all suites)' - - 'Selected variants' + - 'Selected builds' flavour: description: 'Kernel flavour to release in every suite it targets (Selected flavour)' required: true default: 'qcom-next' type: string - kernel-variants: - description: 'Comma-separated variants to release, e.g. qcom-next-trixie,qcom-next-forky (Selected variants)' + builds: + description: 'Comma-separated build names, e.g. qcom-next-trixie,qcom-next-forky (Selected builds)' required: true default: 'qcom-next-trixie' type: string @@ -60,7 +60,7 @@ jobs: env: RELEASE_SCOPE_INPUT: ${{ github.event.inputs.release-scope }} INPUT_FLAVOUR: ${{ github.event.inputs.flavour }} - INPUT_VARIANTS: ${{ github.event.inputs.kernel-variants }} + INPUT_BUILDS: ${{ github.event.inputs.builds }} run: | set -euo pipefail @@ -72,8 +72,8 @@ jobs: args=(--type Release --flavour "$INPUT_FLAVOUR") ;; "Selected variants") - echo "Source: ci/build-matrix.yaml (variants: $INPUT_VARIANTS)" - args=(--type Release --kernel-variant "$INPUT_VARIANTS") + echo "Source: ci/build-matrix.yaml (builds: $INPUT_BUILDS)" + args=(--type Release --build "$INPUT_BUILDS") ;; *) echo "ERROR: unsupported release scope: $RELEASE_SCOPE" >&2 @@ -93,7 +93,7 @@ jobs: # release path (lib/release) instead of the S3 publish path. # --------------------------------------------------------------------------- build: - name: ${{ matrix.kernel_variant }} + name: ${{ matrix.name }} needs: configure-matrix strategy: fail-fast: false @@ -102,7 +102,7 @@ jobs: uses: ./.github/workflows/build-kernel-deb.yml with: suite: ${{ matrix.suite }} - kernel-variant: ${{ matrix.kernel_variant }} + build: ${{ matrix.name }} flavour: ${{ matrix.flavour }} git-clone: ${{ matrix.git_clone }} kernel-branch: ${{ matrix.branch_or_tag }} diff --git a/README.md b/README.md index ca9ab023..3dacc10e 100644 --- a/README.md +++ b/README.md @@ -30,8 +30,7 @@ into LOCALVERSION, so each produces a distinct kernel release (`+qcom-next--g` and `+qcom-next-debug--g`) and therefore a distinct versioned image package that can be installed alongside the other. See [docs/version.md](docs/version.md) for how the version strings are composed. -The flavour is what the kernel is; `kernel_variant` is only what CI calls the -build. +The flavour is what the kernel is; a build's `name` is only what CI calls it. `ci/build-matrix.yaml` is the source of truth; this table is a summary. @@ -42,15 +41,15 @@ Two entry points use the same reusable build pipeline: - **Release** uses a pinned matrix ref and promotes successful Debian packages to the selected production Debusine workspace. -One entry in `deliveries` is one generated package: a single `kernel_variant`, -a single `type`, and a single `suite`. The `kernel_variant` names that one -build and nothing else — it is the Actions job name and what a manual dispatch -asks for. Nothing is expanded or derived at resolve time, so what an entry says -is what gets built: +One entry in `builds` is one generated package: a single `name`, a single +`type`, and a single `suite`. The `name` labels that one build and nothing +else — it is the Actions job name and what a manual dispatch asks for. Nothing +is expanded or derived at resolve time, so what an entry says is what gets +built: ```yaml -deliveries: - - kernel_variant: qcom-next-trixie +builds: + - name: qcom-next-trixie type: Daily suite: trixie flavour: qcom-next @@ -68,7 +67,7 @@ deliveries: - audioreach debian_revision: '0qli1~bpo13+1~' - - kernel_variant: qcom-next-trixie + - name: qcom-next-trixie type: Release suite: trixie flavour: qcom-next @@ -134,8 +133,8 @@ Daily is the recurring build and artifact-publication path. - The scheduled run resolves the full `Daily` matrix. - A manual run selects one **Build scope**: - **Full matrix** builds every configured variant and suite. - - **Selected variants** builds a comma-separated list of `kernel_variant` - names, e.g. `qcom-next-trixie,qcom-next-debug-forky`. + - **Selected builds** builds a comma-separated list of build names, e.g. + `qcom-next-trixie,qcom-next-debug-forky`. - **Selected flavour (all suites)** builds one flavour in every suite it targets. - `latest_tag` resolves the newest matching dated tag; `branch_tip` resolves @@ -152,8 +151,8 @@ Release is the controlled promotion path. - It is manual only and uses one **Release scope** for a kernel variant: - **Selected flavour (all suites)** is the normal release action and promotes every configured Release suite for that flavour. - - **Selected variants** promotes a comma-separated list of `kernel_variant` - names when a targeted action is required. + - **Selected builds** promotes a comma-separated list of build names when a + targeted action is required. - It uses the pinned `branch_or_tag` from the selected `Release` matrix row; it never resolves a newest tag. - Debian source and binary artifacts are built in per-flavour, per-suite @@ -171,15 +170,15 @@ production release controls. ## Matrix Model -`ci/build-matrix.yaml` is a mapping with exactly one top-level key, -`deliveries`. One entry in it is one generated package, so there is no +`ci/build-matrix.yaml` is a mapping with exactly one top-level key, `builds`. +One entry in it is one generated package, so there is no expansion step: `ci/scripts/resolve-matrix.py` validates the whole document, selects the entries matching the requested type, variant, and suite, and hands them to the workflow matrix as they stand. Each entry carries: | Field | Purpose | | --- | --- | -| `kernel_variant` | The name of this one build, and nothing else: its Actions job name, and what a manual dispatch asks for. Unique within a delivery type, so a Daily and its Release may share a name. Never reaches a package name, a version, or a published path. Lowercase letters, digits, and internal hyphens only. | +| `name` | The name of this one build, and nothing else: its Actions job name, and what a manual dispatch asks for. Unique within a delivery type, so a Daily and its Release may share a name. Never reaches a package name, a version, or a published path. Lowercase letters, digits, and internal hyphens only. | | `type` | `Daily` or `Release`. | | `suite` | The one suite this entry builds for. | | `flavour` | The kernel's own identity, and the only matrix field that reaches the built kernel. `derive-localversion.sh` makes it the LOCALVERSION suffix, so two flavours built from one ref get distinct kernel releases and their `linux-image` packages coexist. All entries sharing a flavour build the same package for different suites. | @@ -210,9 +209,9 @@ Because entries are written out in full, the resolver also checks the invariants that span them, which is what makes the duplication safe to read at face value: -- No two entries of one `type` share a `kernel_variant` — a variant names - exactly one build, so a run cannot produce two jobs with one name and a - dispatch cannot be ambiguous. +- No two entries of one `type` share a `name` — a name identifies exactly one + build, so a run cannot produce two jobs with one name and a dispatch cannot + be ambiguous. - A flavour's entries agree on `srcpkg`, `binpkg` and `kernel_config`; those decide what the package *is*, and the entries differ only in where it goes. - A flavour's entries for one suite agree on `dkms`. The module set depends on @@ -236,8 +235,8 @@ debian_revision`) and builds at the revision the daily build would have used. Each entry has a distinct prepared-source artifact, Debusine child workspace, and S3 path keyed by `flavour + suite`. This prevents two flavours that both build, for example, `trixie` from consuming or publishing each other's inputs -or outputs. They key on `flavour`, not `kernel_variant`, so renaming a build -leg never moves a published artifact. +or outputs. They key on `flavour`, not on the build's `name`, so renaming a +build never moves a published artifact. Daily S3 outputs use these layouts, where `` is `-`: @@ -448,7 +447,7 @@ Use **Actions** → **build-kernel-deb** for a one-off build. It is an explicit override workflow, not a matrix-derived delivery flow: use `daily.yml` and `release.yml` for normal Daily and Release operations. -`kernel-variant`, `suite`, and `ref-strategy` are the required build selection. +`build`, `suite`, and `ref-strategy` are the required build selection. All remaining package, configuration, and PR inputs are advanced overrides for validation or debugging. Variant and suite are free-text matrix values rather than static dropdowns, so adding a matrix entry never requires editing the @@ -458,7 +457,7 @@ The available inputs are: | Input | Default | Purpose | | --- | --- | --- | -| `kernel-variant` | `qcom-next-trixie` | Names this build; also selects the matrix entry a blank `debian-revision` is taken from. | +| `build` | `qcom-next-trixie` | Names this build; also selects the matrix entry a blank `debian-revision` is taken from. | | `flavour` | `qcom-next` | Kernel flavour: the LOCALVERSION suffix, and so the kernel release identity. | | `suite` | `trixie` | Target suite. | | `ref-strategy` | `latest_tag` | `latest_tag`, `branch_tip`, or `pinned_ref`. | @@ -505,10 +504,10 @@ and keeps production approval controls in the workflow path. To add a kernel variant: -1. Add one entry to `deliveries` per package the variant should produce: one - per Daily suite and one per Release suite, each spelling out all of its own - fields, and each with a `kernel_variant` unique within its delivery type. - Do not rely on another variant's values. +1. Add one entry to `builds` per package the variant should produce: one per + Daily suite and one per Release suite, each spelling out all of its own + fields, and each with a `name` unique within its delivery type. Do not rely + on another variant's values. 2. Give them all the same `flavour`, distinct from every other flavour's — it becomes the LOCALVERSION suffix, so this is what lets the new kernel install alongside the existing ones. Keep `srcpkg`, `binpkg` and `kernel_config` diff --git a/ci/build-matrix.yaml b/ci/build-matrix.yaml index 3d5bd768..67ab4286 100644 --- a/ci/build-matrix.yaml +++ b/ci/build-matrix.yaml @@ -3,8 +3,8 @@ # # Kernel delivery matrix. # -# One entry in "deliveries" is one generated package: a single kernel_variant, -# a single delivery type, and a single suite. The only list-valued fields are +# One entry in "builds" is one generated package: a single name, a single +# delivery type, and a single suite. The only list-valued fields are # kernel_config and dkms, which name the fragments and the out-of-tree modules # that go into that one package, so an entry needs no expansion — ci/scripts/ # resolve-matrix.py validates the document, filters it, and hands the matching @@ -21,9 +21,10 @@ # identity: it becomes the LOCALVERSION suffix, so qcom-next and # qcom-next-debug produce distinct kernel releases that install alongside each # other, and every suite building one flavour produces the same kernel release. -# kernel_variant names the build in CI - the Actions job, the prepared-source -# artifact, the Debusine workspace, the S3 path - and never reaches a package -# name or version. +# A build's name is what CI calls it: the Actions job name, and what a manual +# dispatch asks for by name. It never reaches a package name or version, and +# never a published path. It must be unique within a delivery type, so a Daily +# and the Release that supersedes it share one. # # dkms varies by suite because it depends on which -dkms packages the # target archive carries: the Debian suites bundle camx and iris-vpu, which @@ -35,11 +36,11 @@ # the ordering discussion in README.md). No two entries may produce the same # srcpkg at the same debian_revision. -deliveries: +builds: # --------------------------------------------------------------------------- # qcom-next — the standard kernel. # --------------------------------------------------------------------------- - - kernel_variant: qcom-next-trixie + - name: qcom-next-trixie type: Daily suite: trixie flavour: qcom-next @@ -57,7 +58,7 @@ deliveries: - audioreach debian_revision: '0qli1~bpo13+1~' - - kernel_variant: qcom-next-forky + - name: qcom-next-forky type: Daily suite: forky flavour: qcom-next @@ -75,7 +76,7 @@ deliveries: - audioreach debian_revision: '0qli1~' - - kernel_variant: qcom-next-resolute + - name: qcom-next-resolute type: Daily suite: resolute flavour: qcom-next @@ -91,7 +92,7 @@ deliveries: - kgsl debian_revision: '0qli1~26.04.1~' - - kernel_variant: qcom-next-trixie + - name: qcom-next-trixie type: Release suite: trixie flavour: qcom-next @@ -109,7 +110,7 @@ deliveries: debian_revision: '0qli1~bpo13+1' target_workspace: qli - - kernel_variant: qcom-next-forky + - name: qcom-next-forky type: Release suite: forky flavour: qcom-next @@ -131,7 +132,7 @@ deliveries: # qcom-next-debug — the same kernel ref built with the in-tree debug # fragments, packaged separately so it installs alongside qcom-next. # --------------------------------------------------------------------------- - - kernel_variant: qcom-next-debug-trixie + - name: qcom-next-debug-trixie type: Daily suite: trixie flavour: qcom-next-debug @@ -151,7 +152,7 @@ deliveries: - audioreach debian_revision: '0qli1~bpo13+1~' - - kernel_variant: qcom-next-debug-forky + - name: qcom-next-debug-forky type: Daily suite: forky flavour: qcom-next-debug @@ -171,7 +172,7 @@ deliveries: - audioreach debian_revision: '0qli1~' - - kernel_variant: qcom-next-debug-trixie + - name: qcom-next-debug-trixie type: Release suite: trixie flavour: qcom-next-debug @@ -191,7 +192,7 @@ deliveries: debian_revision: '0qli1~bpo13+1' target_workspace: qli - - kernel_variant: qcom-next-debug-forky + - name: qcom-next-debug-forky type: Release suite: forky flavour: qcom-next-debug diff --git a/ci/scripts/resolve-matrix.py b/ci/scripts/resolve-matrix.py index abe80be2..47d5cb90 100755 --- a/ci/scripts/resolve-matrix.py +++ b/ci/scripts/resolve-matrix.py @@ -3,8 +3,8 @@ # SPDX-License-Identifier: BSD-3-Clause-Clear """Validate and select entries from the kernel delivery matrix. -ci/build-matrix.yaml holds one entry per generated package: a single -kernel_variant, a single delivery type, and a single suite. Nothing here +ci/build-matrix.yaml holds one entry per generated package: a single named +build, one delivery type, and one suite. Nothing here expands or derives anything -- the matrix is already flat, and each entry states its own debian_revision. This script validates the whole document, selects the entries a caller asked for, and prints them. @@ -15,18 +15,18 @@ Usage: ci/scripts/resolve-matrix.py --type Daily - ci/scripts/resolve-matrix.py --type Daily --kernel-variant qcom-next-trixie - ci/scripts/resolve-matrix.py --type Daily --kernel-variant qcom-next-trixie,qcom-next-forky + ci/scripts/resolve-matrix.py --type Daily --build qcom-next-trixie + ci/scripts/resolve-matrix.py --type Daily --build qcom-next-trixie,qcom-next-forky ci/scripts/resolve-matrix.py --type Release --flavour qcom-next - ci/scripts/resolve-matrix.py --type Daily --kernel-variant qcom-next-trixie \\ + ci/scripts/resolve-matrix.py --type Daily --build qcom-next-trixie \\ --field debian_revision Options: --type TYPE Delivery type to select (Daily or Release). Required. - --kernel-variant VARIANTS Select only these kernel variants, comma-separated. - A variant names one leg, so this is the way to - ask for a specific set of builds. + --build NAMES Select only these builds by name, comma-separated. + A name identifies one build, so this is the way + to ask for a specific set of them. --flavour FLAVOURS Select only these flavours, comma-separated. A flavour spans its suites, so this asks for one kernel everywhere it is built. @@ -39,7 +39,7 @@ Filters combine: an entry must match every filter given. Every name in a filter must match at least one entry of the selected type, so a typo or a - stale variant fails instead of quietly narrowing the build set. + stale name fails instead of quietly narrowing the build set. Output: Without --field, a compact JSON array of the selected entries, ready for a @@ -84,7 +84,7 @@ } REQUIRED_STRING_FIELDS = ( - "kernel_variant", + "name", "type", "suite", "flavour", @@ -108,12 +108,12 @@ REQUIRED_STRING_FIELDS + OPTIONAL_STRING_FIELDS + ("kernel_config", "dkms") ) -# Everything about what is built is anchored on flavour, not kernel_variant. -# The flavour is the kernel's own identity: it becomes the LOCALVERSION +# Everything about what is built is anchored on flavour, not on the build's +# name. The flavour is the kernel's own identity: it becomes the LOCALVERSION # suffix, so two flavours built from one ref produce distinct kernel releases -# whose linux-image packages install alongside each other. kernel_variant -# names the build in CI -- job names, artifacts, Debusine workspaces, S3 -# paths -- and carries no packaging meaning. +# whose linux-image packages install alongside each other. A build's name only +# labels it in CI -- the Actions job, and what a dispatch asks for -- and +# carries no packaging meaning. # Fields that identify the flavour itself rather than one of its build legs. # Every entry for a flavour must agree on them, because they decide what the @@ -240,7 +240,7 @@ def check_entry(entry, report): for field in sorted(set(entry) - KNOWN_FIELDS): report(f"unknown field {field}") - for field in ("kernel_variant", "suite", "flavour"): + for field in ("name", "suite", "flavour"): value = entry.get(field) if isinstance(value, str) and not NAME_RE.match(value): report(f"{field} must use lowercase letters, digits, and internal hyphens") @@ -296,21 +296,21 @@ def describe(entry, index): return f"entry {index}" parts = [ str(entry[field]) - for field in ("kernel_variant", "type", "suite") + for field in ("name", "type", "suite") if isinstance(entry.get(field), str) ] return f"entry {index} ({'/'.join(parts)})" if parts else f"entry {index}" -def check_consistency(deliveries, errors): +def check_consistency(builds, errors): """Validate the invariants that span entries. Entries are written out in full, so the matrix can state a flavour twice and disagree with itself. These checks are what makes that duplication safe to read at face value. - Everything about the package is grouped by flavour. kernel_variant is - only checked for the one thing it is used for: naming a build leg + Everything about the package is grouped by flavour. A build's name is + only checked for the one thing it is used for: identifying that build uniquely. """ by_leg = defaultdict(list) @@ -320,7 +320,7 @@ def check_consistency(deliveries, errors): by_package_version = defaultdict(list) flavours_by_package = defaultdict(set) - for entry in deliveries: + for entry in builds: if not isinstance(entry, dict): continue flavour = entry.get("flavour") @@ -332,7 +332,7 @@ def check_consistency(deliveries, errors): by_flavour[flavour].append(entry) by_flavour_type[(flavour, delivery_type)].append(entry) by_flavour_suite[(flavour, suite)].append(entry) - by_leg[(delivery_type, entry.get("kernel_variant"))].append(entry) + by_leg[(delivery_type, entry.get("name"))].append(entry) for field in ("srcpkg", "binpkg"): if isinstance(entry.get(field), str) and entry[field]: @@ -343,19 +343,19 @@ def check_consistency(deliveries, errors): ): by_package_version[(entry["srcpkg"], entry["debian_revision"])].append(entry) - # kernel_variant names one build leg: it is the Actions job name and what a - # workflow dispatch asks for by name. It only has to be unique within a - # delivery type, because a run only ever resolves one type, and that lets a - # Daily and its Release share a name. Two entries sharing both would give a - # run two identically named jobs and make the dispatch filter ambiguous. - for (delivery_type, variant), entries in sorted( + # A name identifies one build: it is the Actions job name and what a + # workflow dispatch asks for. It only has to be unique within a delivery + # type, because a run only ever resolves one type, and that lets a Daily and + # its Release share a name. Two entries sharing both would give a run two + # identically named jobs and make the dispatch filter ambiguous. + for (delivery_type, name), entries in sorted( by_leg.items(), key=lambda item: str(item[0]) ): if len(entries) > 1: suites = ", ".join(sorted(str(e.get("suite")) for e in entries)) errors.append( - f"kernel_variant {variant} is used by {len(entries)} {delivery_type} " - f"entries (suites {suites}); a variant names exactly one build" + f"name {name} is used by {len(entries)} {delivery_type} entries " + f"(suites {suites}); a name identifies exactly one build" ) for flavour, entries in sorted(by_flavour.items()): @@ -436,21 +436,21 @@ def check_consistency(deliveries, errors): ) -def validate(deliveries): +def validate(builds): """Return every problem found in the matrix, as a list of messages.""" errors = [] - for index, entry in enumerate(deliveries): + for index, entry in enumerate(builds): if not isinstance(entry, dict): errors.append(f"{describe(entry, index)}: delivery entries must be mappings") continue label = describe(entry, index) check_entry(entry, lambda message, label=label: errors.append(f"{label}: {message}")) - check_consistency(deliveries, errors) + check_consistency(builds, errors) return errors def load_matrix(path): - """Read, parse, and validate the matrix, returning its deliveries.""" + """Read, parse, and validate the matrix, returning its builds.""" try: with open(path, encoding="utf-8") as handle: document = yaml.safe_load(handle) @@ -462,32 +462,32 @@ def load_matrix(path): sys.exit(f"ERROR: Invalid YAML in {path}: {error}") if not isinstance(document, dict): - sys.exit(f"ERROR: {path}: matrix root must be a mapping with a deliveries key") + sys.exit(f"ERROR: {path}: matrix root must be a mapping with a builds key") - # deliveries is the whole schema. Anything else at the root is a leftover - # from an older matrix (suite_suffix_mapping, say) that would otherwise sit + # builds is the whole schema. Anything else at the root is a leftover from + # an older matrix (suite_suffix_mapping, say) that would otherwise sit # there looking authoritative while nothing read it. - unknown_root = sorted(set(document) - {"deliveries"}) + unknown_root = sorted(set(document) - {"builds"}) if unknown_root: sys.exit( f"ERROR: {path}: unknown top-level key(s) {', '.join(unknown_root)}; " - "deliveries is the only one" + "builds is the only one" ) - deliveries = document.get("deliveries") - if not isinstance(deliveries, list): - sys.exit(f"ERROR: {path}: deliveries must be a list") - if not deliveries: - sys.exit(f"ERROR: {path}: deliveries must contain at least one entry") + builds = document.get("builds") + if not isinstance(builds, list): + sys.exit(f"ERROR: {path}: builds must be a list") + if not builds: + sys.exit(f"ERROR: {path}: builds must contain at least one entry") - errors = validate(deliveries) + errors = validate(builds) if errors: sys.exit( f"ERROR: Invalid kernel delivery matrix in {path}:\n" + "\n".join(f" - {error}" for error in errors) ) - return deliveries + return builds def parse_filter(value): @@ -497,7 +497,7 @@ def parse_filter(value): return [name.strip() for name in value.split(",") if name.strip()] -def select(deliveries, delivery_type, filters): +def select(builds, delivery_type, filters): """Return the entries matching the requested filters, in matrix order. filters maps a field name to the list of values allowed for it, or to @@ -505,7 +505,7 @@ def select(deliveries, delivery_type, filters): """ return [ entry - for entry in deliveries + for entry in builds if entry["type"] == delivery_type and all( wanted is None or entry[field] in wanted @@ -514,14 +514,14 @@ def select(deliveries, delivery_type, filters): ] -def unmatched_filters(deliveries, delivery_type, filters): +def unmatched_filters(builds, delivery_type, filters): """Names asked for that no entry of this delivery type offers.""" missing = [] for field, wanted in filters.items(): if wanted is None: continue available = { - entry[field] for entry in deliveries if entry["type"] == delivery_type + entry[field] for entry in builds if entry["type"] == delivery_type } for name in wanted: if name not in available: @@ -561,9 +561,9 @@ def main(): "--type", required=True, choices=DELIVERY_TYPES, help="delivery type to select" ) parser.add_argument( - "--kernel-variant", + "--build", default="", - help="select only these kernel variants, comma-separated", + help="select only these builds by name, comma-separated", ) parser.add_argument( "--flavour", default="", help="select only these flavours, comma-separated" @@ -581,25 +581,25 @@ def main(): ) args = parser.parse_args() - deliveries = load_matrix(args.matrix_file) + builds = load_matrix(args.matrix_file) filters = { - "kernel_variant": parse_filter(args.kernel_variant), + "name": parse_filter(args.build), "flavour": parse_filter(args.flavour), "suite": parse_filter(args.suite), } what = describe_selection(args.type, filters) # Report a name that matches nothing before reporting an empty selection: - # "no Daily entry has kernel_variant qcom-next" says what to fix, where + # "no Daily entry has name qcom-next" says what to fix, where # "no entries found" leaves the reader to work out which filter was wrong. - unmatched = unmatched_filters(deliveries, args.type, filters) + unmatched = unmatched_filters(builds, args.type, filters) if unmatched: sys.exit( f"ERROR: Nothing to build for {what}:\n" + "\n".join(f" - {problem}" for problem in unmatched) ) - selected = select(deliveries, args.type, filters) + selected = select(builds, args.type, filters) if not selected: sys.exit(f"ERROR: No matrix entries found for {what}") @@ -610,7 +610,7 @@ def main(): if len(selected) > 1: sys.exit( f"ERROR: --field {args.field} needs exactly one entry, but {what} " - f"selects {len(selected)}; narrow it with --kernel-variant and --suite" + f"selects {len(selected)}; narrow it with --build" ) entry = for_workflow(selected[0]) From 400e951a9f9ca4885abc107af042a0d4304e501c Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 08/59] ci: add the mainline, linux-next and Arduino flavours Three kernels join the delivery matrix, each built for trixie and forky: mainline from the tip of Linus's tree, next from the newest next-YYYYMMDD tag of linux-next, and qcom-arduino from the early/hwe/arduino topic branch of kernel-topics. mainline and next give early warning of upstream breakage; qcom-arduino delivers hardware enablement that has not reached qcom-next. All three track something moving - two branch tips and a tag cut every night - so none of them has a Release entry: a Release must name an immutable ref, and neither a branch tip nor tonight's tag is one. They are built and published daily and never promoted to qli. That makes the "every flavour defines at least one of each delivery type" invariant wrong, so it becomes the half that carries the meaning: a Release entry requires the Daily that tests it, but a Daily needs no Release. The symmetric rule was not protecting anything - a flavour built daily and never promoted is a coherent thing to configure, and the rule only prevented saying so. mainline and next bundle no DKMS modules. kgsl, camx and iris-vpu are built against the Qualcomm tree, and a listed module is a presence contract: naming one whose BUILD_EXCLUSIVE gates exclude the kernel fails the build rather than shipping without it. qcom-arduino is a Qualcomm branch and carries the same three its qcom-next siblings do on the Debian suites. Signed-off-by: Christopher Obbard --- README.md | 23 +++++--- ci/build-matrix.yaml | 108 +++++++++++++++++++++++++++++++++++ ci/scripts/resolve-matrix.py | 14 +++-- 3 files changed, 132 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 3dacc10e..eca38a6d 100644 --- a/README.md +++ b/README.md @@ -24,13 +24,20 @@ kernel flavour owns as many entries as it has suites and delivery types. |---------|----------------|-------------------|--------------|----------------|-------| | `qcom-next` | `linux-qcom-next` | `linux-image-qcom-next` | trixie, forky, resolute | trixie, forky | Standard kernel | | `qcom-next-debug` | `linux-qcom-next-debug` | `linux-image-qcom-next-debug` | trixie, forky | trixie, forky | Adds `arch/arm64/configs/qcom_debug.config` and `kernel/configs/debug.config` from the kernel source, via `intree:` entries | - -Both build the same kernel ref. `derive-localversion.sh` folds the *flavour* -into LOCALVERSION, so each produces a distinct kernel release -(`+qcom-next--g` and `+qcom-next-debug--g`) and therefore a -distinct versioned image package that can be installed alongside the other. See +| `qcom-arduino` | `linux-qcom-arduino` | `linux-image-qcom-arduino` | trixie, forky | none | Arduino hardware-enablement topic branch (`early/hwe/arduino` of `kernel-topics`) | +| `mainline` | `linux-mainline` | `linux-image-mainline` | trixie, forky | none | Tip of Linus's tree, tracked for early warning of upstream breakage. No DKMS modules | +| `next` | `linux-next` | `linux-image-next` | trixie, forky | none | Newest `next-YYYYMMDD` tag of linux-next. No DKMS modules | + +`derive-localversion.sh` folds the *flavour* into LOCALVERSION, so each +produces a distinct kernel release (`+qcom-next--g`, +`+qcom-next-debug--g`, and so on) and therefore a distinct versioned +image package that can be installed alongside the others. The flavour is what +the kernel is; a build's `name` is only what CI calls it. See [docs/version.md](docs/version.md) for how the version strings are composed. -The flavour is what the kernel is; a build's `name` is only what CI calls it. + +The last three track a moving upstream — a branch tip, or a tag cut every +night — and so have no `Release` entries: a release must name an immutable ref +to promote. They are built and published daily and never promoted to `qli`. `ci/build-matrix.yaml` is the source of truth; this table is a summary. @@ -221,7 +228,9 @@ face value: - A flavour's entries of one `type` agree on `git_clone`, `branch_or_tag`, `ref_strategy` and `tag_pattern`, so a forgotten suite cannot quietly ship a different kernel from its siblings after a release ref bump. -- Every flavour defines at least one `Daily` and one `Release` entry. +- A flavour with a `Release` entry also has a `Daily` one, so nothing is + promoted that the daily build has not tested. The converse is allowed: a + flavour tracking a moving upstream is built daily and never released. - No `srcpkg` or `binpkg` is shared between flavours, and no two entries build the same `srcpkg` at the same `debian_revision`. - Where a suite has both, its `Daily` revision is its `Release` revision plus a diff --git a/ci/build-matrix.yaml b/ci/build-matrix.yaml index 67ab4286..c668faeb 100644 --- a/ci/build-matrix.yaml +++ b/ci/build-matrix.yaml @@ -30,6 +30,11 @@ # target archive carries: the Debian suites bundle camx and iris-vpu, which # Ubuntu does not package. # +# A flavour tracking a moving upstream is Daily only. A Release entry must name +# an immutable ref, and a branch tip or a tag cut every night is not one, so +# mainline, next and the topic branches are built and published daily and never +# promoted. A Release entry still requires the Daily that tests it. +# # debian_revision is stated outright, not derived. Daily revisions end in "~" # so they sort below the Release revision for the same suite; the rest of the # revision places the suite where it belongs relative to the other suites (see @@ -211,3 +216,106 @@ builds: - audioreach debian_revision: '0qli1' target_workspace: qli + + # --------------------------------------------------------------------------- + # qcom-arduino — the Arduino hardware-enablement topic branch. + # + # Daily only: a topic branch has no immutable ref to promote from. + # --------------------------------------------------------------------------- + - name: qcom-arduino-trixie + type: Daily + suite: trixie + flavour: qcom-arduino + git_clone: https://github.com/qualcomm-linux/kernel-topics + branch_or_tag: early/hwe/arduino + ref_strategy: branch_tip + srcpkg: linux-qcom-arduino + binpkg: linux-image-qcom-arduino + kernel_config: [] + dkms: + - kgsl + - camx + - iris-vpu + debian_revision: '0qli~bpo13+1~' + + - name: qcom-arduino-forky + type: Daily + suite: forky + flavour: qcom-arduino + git_clone: https://github.com/qualcomm-linux/kernel-topics + branch_or_tag: early/hwe/arduino + ref_strategy: branch_tip + srcpkg: linux-qcom-arduino + binpkg: linux-image-qcom-arduino + kernel_config: [] + dkms: + - kgsl + - camx + - iris-vpu + debian_revision: '0qli~' + + # --------------------------------------------------------------------------- + # mainline — Linus's tree, tracked for early warning of upstream breakage. + # + # Daily only, and no DKMS: the out-of-tree modules target the Qualcomm tree, + # and a listed module is a presence contract that would fail the build here. + # --------------------------------------------------------------------------- + - name: mainline-trixie + type: Daily + suite: trixie + flavour: mainline + git_clone: https://github.com/torvalds/linux + branch_or_tag: master + ref_strategy: branch_tip + srcpkg: linux-mainline + binpkg: linux-image-mainline + kernel_config: [] + dkms: [] + debian_revision: '0qli~bpo13+1~' + + - name: mainline-forky + type: Daily + suite: forky + flavour: mainline + git_clone: https://github.com/torvalds/linux + branch_or_tag: master + ref_strategy: branch_tip + srcpkg: linux-mainline + binpkg: linux-image-mainline + kernel_config: [] + dkms: [] + debian_revision: '0qli~' + + # --------------------------------------------------------------------------- + # next — linux-next, the integration tree, one step ahead of mainline. + # + # next-YYYYMMDD tags are cut daily, so latest_tag resolves the newest of + # them the same way qcom-next-* does. Daily only, and no DKMS, as above. + # --------------------------------------------------------------------------- + - name: next-trixie + type: Daily + suite: trixie + flavour: next + git_clone: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git + branch_or_tag: master + ref_strategy: latest_tag + tag_pattern: 'next-*' + srcpkg: linux-next + binpkg: linux-image-next + kernel_config: [] + dkms: [] + debian_revision: '0qli~bpo13+1~' + + - name: next-forky + type: Daily + suite: forky + flavour: next + git_clone: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git + branch_or_tag: master + ref_strategy: latest_tag + tag_pattern: 'next-*' + srcpkg: linux-next + binpkg: linux-image-next + kernel_config: [] + dkms: [] + debian_revision: '0qli~' diff --git a/ci/scripts/resolve-matrix.py b/ci/scripts/resolve-matrix.py index 47d5cb90..3f5082ec 100755 --- a/ci/scripts/resolve-matrix.py +++ b/ci/scripts/resolve-matrix.py @@ -366,13 +366,15 @@ def check_consistency(builds, errors): f"flavour {flavour} must use one {field} across all its " "entries (got " + ", ".join(sorted(values)) + ")" ) + # A Release must be preceded by the Daily that tests it, but the + # converse does not hold: a flavour tracking a moving upstream has + # nothing immutable to pin, so it is built daily and never promoted. types = {entry.get("type") for entry in entries} - for delivery_type in DELIVERY_TYPES: - if delivery_type not in types: - errors.append( - f"flavour {flavour} has no {delivery_type} entry; " - "every flavour must define at least one of each" - ) + if "Release" in types and "Daily" not in types: + errors.append( + f"flavour {flavour} has a Release entry but no Daily entry; " + "a release must be preceded by the daily build that tests it" + ) for (flavour, suite), entries in sorted( by_flavour_suite.items(), key=lambda item: str(item[0]) From 571e17c79fffff9a3cbd900cf49f2114b98c2b44 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 09/59] ci: drop the resolve job and gate the build legs directly The resolve job took a whole ubuntu-latest runner to turn inputs.suite into the string "debian" or "ubuntu" via a static case statement, which the two build legs then compared against. A job-level if: expression answers the same question for free, so every run paid for a runner spin-up and a queue wait ahead of both legs to learn something already fixed at dispatch time. Both legs now test the suite themselves, the Ubuntu one negating what the Debian one asserts, so the two cannot drift into an overlap or a gap. The list is spelled out twice because GitHub Actions does not expand YAML anchors and there is no other way to share it without reintroducing a job. The trixie default is unchanged. One behaviour change comes with it: GitHub's contains() compares strings case-insensitively, so a suite of "Trixie" now classifies as debian where the case statement sent it to ubuntu. Neither spelling reaches here from daily.yml or release.yml, whose suites come from the matrix in lowercase. The build-name check moves to the Resolve Debian revision step, the only place the name is consumed, and becomes an ::error:: so it shows in the run summary rather than only in the log. Signed-off-by: Christopher Obbard --- .github/workflows/build-kernel-deb.yml | 43 +++++--------------------- 1 file changed, 8 insertions(+), 35 deletions(-) diff --git a/.github/workflows/build-kernel-deb.yml b/.github/workflows/build-kernel-deb.yml index 07f6a668..184e5896 100644 --- a/.github/workflows/build-kernel-deb.yml +++ b/.github/workflows/build-kernel-deb.yml @@ -166,37 +166,6 @@ permissions: packages: read jobs: - # --------------------------------------------------------------------------- - # resolve: classify the target suite as 'debian' or 'ubuntu'. - # --------------------------------------------------------------------------- - resolve: - name: Resolve suite family - runs-on: ubuntu-latest - outputs: - family: ${{ steps.classify.outputs.family }} - steps: - - name: Classify suite - id: classify - shell: bash - env: - SUITE_INPUT: ${{ inputs.suite || 'trixie' }} - BUILD_INPUT: ${{ inputs.build || 'qcom-next-trixie' }} - run: | - suite="$SUITE_INPUT" - build_name="$BUILD_INPUT" - [[ "$build_name" =~ ^[a-z0-9]+([a-z0-9-]*[a-z0-9])?$ ]] || { - echo "ERROR: build name must use lowercase letters, digits, and internal hyphens" >&2 - exit 1 - } - case "$suite" in - trixie|forky|sid|unstable|bookworm) - family=debian ;; - *) - family=ubuntu ;; - esac - echo "family=$family" >> "$GITHUB_OUTPUT" - echo "Suite '$suite' classified as family=$family" - # --------------------------------------------------------------------------- # prepare: clone kernel source, inject packaging, run prepare-source.sh. # --------------------------------------------------------------------------- @@ -315,6 +284,10 @@ jobs: BUILD_INPUT: ${{ inputs.build || 'qcom-next-trixie' }} run: | set -euo pipefail + [[ "$BUILD_INPUT" =~ ^[a-z0-9]+([a-z0-9-]*[a-z0-9])?$ ]] || { + echo "::error::build name must use lowercase letters, digits, and internal hyphens" + exit 1 + } INPUT_REVISION="$DEBIAN_REVISION_INPUT" if [[ -n "$INPUT_REVISION" ]]; then DEBIAN_REVISION="$INPUT_REVISION" @@ -431,8 +404,8 @@ jobs: # --------------------------------------------------------------------------- debusine-build: name: Build in Debusine - needs: [resolve, prepare] - if: ${{ needs.resolve.outputs.family == 'debian' }} + needs: [prepare] + if: ${{ contains(fromJSON('["trixie", "forky", "sid", "unstable", "bookworm"]'), inputs.suite || 'trixie') }} uses: ./.github/workflows/build-kernel-debusine.yml with: suite: ${{ inputs.suite || 'trixie' }} @@ -450,8 +423,8 @@ jobs: # --------------------------------------------------------------------------- ubuntu-build: name: Build (Ubuntu, docker) - needs: [resolve, prepare] - if: ${{ needs.resolve.outputs.family == 'ubuntu' }} + needs: [prepare] + if: ${{ !contains(fromJSON('["trixie", "forky", "sid", "unstable", "bookworm"]'), inputs.suite || 'trixie') }} uses: ./.github/workflows/build-kernel-ubuntu.yml with: distro: ${{ inputs.suite || 'resolute' }} From f55c3646a923fba50f76e1d6ac610c4907696a0f Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 10/59] ci: branch on the build path in the caller, not inside the build A full daily run rendered 21 skipped jobs against 11 real build legs, and a PR the same. Every leg started both family backends and skipped one; every Debian leg started both Debusine tails and skipped one. Release was worse per leg: all four of its legs are Debian, so the Ubuntu backend never once ran. Both branches were decided by data the caller already knew. A suite's family follows from the suite, which the matrix states; publish-versus-promote is fixed by which workflow is running, since a daily never promotes and a release always does. Passing both down into one shared workflow and re-deciding them there with if: is what minted the skipped jobs, and GitHub offers no way to take that back: a job is conditional or it is not, an empty dynamic matrix is an error rather than an empty job list, and the guard that avoids the error is another skipped job. So decide in the caller and give each decision its own workflow. resolve-matrix.py derives a family from each entry's suite and filters on it, daily.yml and pr-build.yml split one selection into a debian and an ubuntu matrix, and each leg calls only the workflow that builds it. build-kernel-deb.yml, which existed to make the choice, is gone. The two Debian workflows are the same build with different tails -- build-kernel-debian.yml publishes to S3, release-kernel-debian.yml promotes to a target workspace -- so neither carries the other's tail as a job to skip. Release entries only ever reach the second: promotion runs through Debusine, which builds no Ubuntu suite, so resolve-matrix.py now rejects a Release entry for one. That was previously accepted and would have built the package, taken the Ubuntu path, published to the daily S3 location and reported success without releasing anything. The shared steps move into two composite actions rather than a common parent workflow. A reusable workflow would have put its name into every job's, which is the nesting that "qcom-next-resolute / Build (Ubuntu, docker) / Build kernel package" already suffered from; a composite action adds no segment. That name is now "qcom-next-resolute / Build kernel package", and a full daily run is 32 jobs across 11 legs with nothing skipped. Suite stays a free-text dispatch input, so a new suite still needs no workflow edit. Since the workflow now implies the family, prepare-kernel-source takes the family its caller builds and rejects a suite belonging to the other before anything is cloned. Its family list is the one resolve-matrix.py routes by. Two dispatch scopes were repaired in passing: daily.yml and release.yml both still matched "Selected variants" in a case whose input has offered "Selected builds" since the matrix keys were renamed, so choosing it failed the run with "unsupported build scope". No build changes what it produces. The job names, S3 paths, Debusine workspaces, revisions and LOCALVERSIONs are all as they were. The staging build-dependency workspaces come along in the same move. They were picked by a ternary on target-workspace, which was the only thing separating a daily run from a release one while both lived in one workflow. That separation is now the choice of workflow, so debusine-build takes the list as an input and each caller states its own: build-kernel-debian.yml reads qli and qli-staging, release-kernel-debian.yml reads qli alone. Same two answers, no longer derived from a field that no longer distinguishes anything. Signed-off-by: Christopher Obbard --- .github/actions/debusine-build/action.yml | 125 +++++ .../actions/prepare-kernel-source/action.yml | 318 +++++++++++++ .github/workflows/build-kernel-deb.yml | 431 ------------------ .github/workflows/build-kernel-debian.yml | 328 +++++++++++++ .github/workflows/build-kernel-debusine.yml | 242 ---------- .github/workflows/build-kernel-ubuntu.yml | 234 ++++++++-- .github/workflows/daily.yml | 77 +++- .github/workflows/pr-build.yml | 59 ++- .github/workflows/release-kernel-debian.yml | 210 +++++++++ .github/workflows/release.yml | 14 +- README.md | 119 +++-- ci/scripts/resolve-matrix.py | 77 +++- 12 files changed, 1461 insertions(+), 773 deletions(-) create mode 100644 .github/actions/debusine-build/action.yml create mode 100644 .github/actions/prepare-kernel-source/action.yml delete mode 100644 .github/workflows/build-kernel-deb.yml create mode 100644 .github/workflows/build-kernel-debian.yml delete mode 100644 .github/workflows/build-kernel-debusine.yml create mode 100644 .github/workflows/release-kernel-debian.yml diff --git a/.github/actions/debusine-build/action.yml b/.github/actions/debusine-build/action.yml new file mode 100644 index 00000000..0eff3ade --- /dev/null +++ b/.github/actions/debusine-build/action.yml @@ -0,0 +1,125 @@ +name: Build in Debusine +description: > + Generate the Debian source package from the prepared tree and submit it to + Debusine, returning the CI workspace it was built in. + +# Shared by the two Debian workflows, which differ only in what they do with +# the result: build-kernel-debian.yml downloads the .deb files and uploads them +# to S3, release-kernel-debian.yml promotes them to a target workspace. Holding +# these steps in a composite action lets each workflow declare its own runner, +# container and environment while keeping one copy of the build itself. +# +# The caller must have checked out pkg-linux-qcom into the workspace first, +# which is also where this action comes from. Runs inside the +# debusine-pkg-builder container. +# +# vars and secrets contexts do not reach a composite action, so the Debusine +# host, scope and credentials come in as inputs from the calling workflow. + +inputs: + suite: + description: 'Target Debian suite (trixie, forky, sid)' + required: true + flavour: + description: 'Kernel flavour, isolating artifacts and Debusine workspaces' + required: true + debusine-host: + description: 'Debusine instance hostname' + required: true + debusine-scope: + description: 'Debusine scope' + required: true + debusine-user: + description: 'Debusine account used to submit the build' + required: true + debusine-token: + description: 'Debusine token for the CI child workspace' + required: true + debusine-parent-workspace: + description: 'Parent Debusine workspace for CI child workspace creation' + default: 'qli-ci' + extra-build-dep-workspaces: + description: > + Workspaces the build resolves its build-dependencies from, space + separated. A staging build may depend on a package version that has only + reached qli-staging; a release build reads qli alone, so a released + kernel is never built against something unreleased. + default: 'qli' + debusine-action-ref: + description: 'Ref of qualcomm-linux/debusine-action to use' + default: 'main' + +outputs: + workspace: + description: 'Debusine CI workspace the package was built in' + value: ${{ steps.build-debusine.outputs.workspace }} + workspace_url: + description: 'Browser URL of that workspace' + value: ${{ steps.build-debusine.outputs.workspace_url }} + srcpkg_version: + description: 'Version of the generated source package' + value: ${{ steps.generate-srcpkg.outputs.srcpkg_version }} + +runs: + using: composite + steps: + - name: Checkout debusine-action helpers + uses: actions/checkout@v5 + with: + repository: qualcomm-linux/debusine-action + ref: ${{ inputs.debusine-action-ref }} + path: debusine-action + fetch-depth: 1 + sparse-checkout: | + lib + + - name: Download prepared source tree + uses: actions/download-artifact@v4 + with: + name: kernel-srcpkg-${{ inputs.flavour }}-${{ inputs.suite }} + path: /tmp/srcpkg-artifact + + - name: Extract prepared source tree + shell: bash + run: | + mkdir srcpkg + tar xzf /tmp/srcpkg-artifact/*.tar.gz \ + -C srcpkg --strip-components=1 + + - name: Generate source package + id: generate-srcpkg + shell: bash + env: + SUITE: ${{ inputs.suite }} + DEBUSINE_ASSEMBLE_ORIG: "true" + run: | + SUITE="$SUITE" debusine-action/lib/generate-source-package + + - name: Build in Debusine + id: build-debusine + shell: bash + env: + GITHUB_REPOSITORY_ID: ${{ github.repository_id }} + GITHUB_RUN_ID: ${{ github.run_id }} + GITHUB_RUN_ATTEMPT: ${{ github.run_attempt }} + JOB_INDEX: ${{ inputs.flavour }}-${{ inputs.suite }} + DEBUSINE_HOST: ${{ inputs.debusine-host }} + DEBUSINE_SCOPE: ${{ inputs.debusine-scope }} + DEBUSINE_USER: ${{ inputs.debusine-user }} + DEBUSINE_TOKEN: ${{ inputs.debusine-token }} + DEBUSINE_PARENT_WORKSPACE: ${{ inputs.debusine-parent-workspace }} + SUITE: ${{ inputs.suite }} + EXTRA_BUILD_DEP_WORKSPACES: ${{ inputs.extra-build-dep-workspaces }} + run: | + # No -x here: DEBUSINE_TOKEN is in env, keep xtrace off. + set -euo pipefail + debusine-action/lib/build + workspace=$(sed -n 's/^workspace=//p' "$GITHUB_OUTPUT") + echo "workspace_url=https://${DEBUSINE_HOST}/${DEBUSINE_SCOPE}/${workspace}/" >> "$GITHUB_OUTPUT" + + - name: Note Debusine workspace URL + shell: bash + env: + WORKSPACE_URL: ${{ steps.build-debusine.outputs.workspace_url }} + run: | + echo "Debusine Workspace URL: $WORKSPACE_URL" >> "$GITHUB_STEP_SUMMARY" diff --git a/.github/actions/prepare-kernel-source/action.yml b/.github/actions/prepare-kernel-source/action.yml new file mode 100644 index 00000000..ae171847 --- /dev/null +++ b/.github/actions/prepare-kernel-source/action.yml @@ -0,0 +1,318 @@ +name: Prepare kernel source +description: > + Resolve the kernel ref, clone it, apply any PR patches, inject debian/ and + generate the source package, then upload the prepared tree as an artifact. + +# Every build takes this path, whichever family it belongs to, so it lives in a +# composite action rather than a reusable workflow: the two family workflows +# call it as a step of their own prepare job, which keeps one copy of these +# steps without adding a nesting level to every job name in the checks list. +# +# The caller must have checked out pkg-linux-qcom into the workspace first: +# debian/ and ci/ come from that checkout, and so does this action. +# +# Runs on the self-hosted arm64 runner. Leaves KERNEL_DIR, KERNEL_REF, +# KERNEL_SHA, LOCALVERSION and DEBIAN_REVISION in the job environment. + +inputs: + build: + description: 'Name of this build, as listed in ci/build-matrix.yaml' + required: true + suite: + description: 'Target suite' + required: true + expect-family: + description: > + Build family the calling workflow builds, debian or ubuntu. The suite is + a free-text matrix value, so this catches one handed to the workflow for + the other family before anything is cloned or built. + required: true + flavour: + description: 'Kernel flavour: the LOCALVERSION suffix and so the kernel release identity' + required: true + kernel-url: + description: 'Kernel repository URL' + default: 'https://github.com/qualcomm-linux/kernel' + ref-strategy: + description: 'Kernel ref strategy: latest_tag, branch_tip, or pinned_ref' + default: 'latest_tag' + kernel-branch: + description: 'Branch for branch_tip or immutable ref for pinned_ref (ignored for latest_tag)' + default: 'qcom-next' + tag-pattern: + description: 'Tag glob used only by latest_tag' + default: 'qcom-next-*' + srcpkg: + description: 'Debian source package name' + default: 'linux-qcom-next' + binpkg: + description: 'Kernel image metapackage name' + default: 'linux-image-qcom-next' + kernel-config: + description: 'Extra config fragments applied on top of all of debian/config-available/' + default: '' + dkms: + description: 'Out-of-tree DKMS modules to bundle, comma-separated and without the -dkms suffix' + default: '' + debian-revision: + description: 'Debian revision override (empty takes the Daily revision the matrix gives this build)' + default: '' + localversion: + description: 'LOCALVERSION override (auto-derived from the resolved ref if empty)' + default: '' + kver-extra: + description: 'Extra package-version suffix (e.g. -ci42)' + default: '' + debug-build: + description: 'Enable the debug build configuration' + default: 'false' + qcom-next-pr: + description: 'qcom-next PR numbers to merge, space-separated' + default: '' + kernel-topics-pr: + description: 'kernel-topics PR numbers to apply as patches, space-separated' + default: '' + +runs: + using: composite + steps: + - name: Resolve build environment + shell: bash + env: + SUITE_INPUT: ${{ inputs.suite }} + EXPECT_FAMILY: ${{ inputs.expect-family }} + run: | + set -euo pipefail + # Family and docker image in one place, because they answer the same + # question about a suite. Debian-family suites build in the trixie + # image: docker-pkg-build and pkg-builder only support trixie for + # Debian, and Debusine handles the actual suite-specific build + # internally. Ubuntu-family suites build in an image of their own + # suite. The family list here is the one in + # ci/scripts/resolve-matrix.py, which routes an entry to the workflow + # that calls this action. + case "$SUITE_INPUT" in + trixie|bookworm) FAMILY=debian; DOCKER_DISTRO="$SUITE_INPUT" ;; + forky|sid|unstable) FAMILY=debian; DOCKER_DISTRO=trixie ;; + *) FAMILY=ubuntu; DOCKER_DISTRO="$SUITE_INPUT" ;; + esac + + if [[ "$FAMILY" != "$EXPECT_FAMILY" ]]; then + echo "::error::Suite $SUITE_INPUT is $FAMILY-family, but this workflow builds the $EXPECT_FAMILY family." + echo "::error::Dispatch the $FAMILY build workflow for it instead." + exit 1 + fi + + echo "DISTRO=$SUITE_INPUT" >> "$GITHUB_ENV" + echo "DOCKER_DISTRO=$DOCKER_DISTRO" >> "$GITHUB_ENV" + echo "Suite $SUITE_INPUT is $FAMILY-family and builds in the $DOCKER_DISTRO image" + + - name: Checkout docker-pkg-build + uses: actions/checkout@v4 + with: + repository: qualcomm-linux/docker-pkg-build + ref: main + path: docker-pkg-build + + - name: Build docker image + shell: bash + run: | + ./docker-pkg-build/docker_deb_build.py --rebuild -d "$DOCKER_DISTRO" + docker image ls + + - name: Resolve kernel ref + shell: bash + env: + KERNEL_URL_INPUT: ${{ inputs.kernel-url }} + BRANCH_INPUT: ${{ inputs.kernel-branch }} + REF_STRATEGY_INPUT: ${{ inputs.ref-strategy }} + TAG_PATTERN_INPUT: ${{ inputs.tag-pattern }} + BUILD_INPUT: ${{ inputs.build }} + run: | + set -euo pipefail + [[ "$BUILD_INPUT" =~ ^[a-z0-9]+([a-z0-9-]*[a-z0-9])?$ ]] || { + echo "::error::build name must use lowercase letters, digits, and internal hyphens" + exit 1 + } + echo "KERNEL_URL=$KERNEL_URL_INPUT" >> "$GITHUB_ENV" + + case "$REF_STRATEGY_INPUT" in + latest_tag) + KERNEL_REF=$(ci/scripts/resolve-kernel-ref.sh \ + --url "$KERNEL_URL_INPUT" \ + --latest-tag "$TAG_PATTERN_INPUT") + ;; + branch_tip|pinned_ref) + KERNEL_REF=$(ci/scripts/resolve-kernel-ref.sh \ + --url "$KERNEL_URL_INPUT" --ref "$BRANCH_INPUT") + ;; + *) + echo "::error::unsupported ref strategy: $REF_STRATEGY_INPUT" + exit 1 + ;; + esac + + echo "KERNEL_REF=$KERNEL_REF" >> "$GITHUB_ENV" + echo "Build: $BUILD_INPUT" + echo "Ref strategy: $REF_STRATEGY_INPUT" + echo "Resolved kernel ref: $KERNEL_REF" + + - name: Clone kernel source + shell: bash + run: | + set -euo pipefail + KERNEL_DIR="$GITHUB_WORKSPACE/kernel-source" + echo "KERNEL_DIR=$KERNEL_DIR" >> "$GITHUB_ENV" + echo "Cloning $KERNEL_URL @ $KERNEL_REF (shallow)..." + git clone --depth 1 --single-branch --branch "$KERNEL_REF" --no-tags \ + "$KERNEL_URL" "$KERNEL_DIR" + KERNEL_SHA=$(git -C "$KERNEL_DIR" rev-parse HEAD) + echo "KERNEL_SHA=$KERNEL_SHA" >> "$GITHUB_ENV" + echo "Kernel HEAD SHA: $KERNEL_SHA" + + - name: Derive LOCALVERSION + shell: bash + env: + LOCALVERSION_INPUT: ${{ inputs.localversion }} + FLAVOUR_INPUT: ${{ inputs.flavour }} + run: | + set -euo pipefail + if [[ -n "$LOCALVERSION_INPUT" ]]; then + echo "Using explicit LOCALVERSION override: $LOCALVERSION_INPUT" + echo "LOCALVERSION=$LOCALVERSION_INPUT" >> "$GITHUB_ENV" + else + LOCALVERSION=$(ci/scripts/derive-localversion.sh \ + --flavour "$FLAVOUR_INPUT" \ + --ref "$KERNEL_REF" \ + --sha "$(echo "$KERNEL_SHA" | cut -c1-12)") + echo "LOCALVERSION=$LOCALVERSION" >> "$GITHUB_ENV" + echo "Derived LOCALVERSION: $LOCALVERSION" + fi + + - name: Resolve Debian revision + # Callers driven by the matrix pass the entry's own debian-revision + # straight through. A direct workflow_dispatch that leaves it empty looks + # the same entry up itself, so a manual build of a configured build gets + # exactly the revision the daily build would have used. The lookup is + # always the Daily entry: direct dispatch is build-only and never + # promotes. + shell: bash + env: + DEBIAN_REVISION_INPUT: ${{ inputs.debian-revision }} + BUILD_INPUT: ${{ inputs.build }} + run: | + set -euo pipefail + if [[ -n "$DEBIAN_REVISION_INPUT" ]]; then + DEBIAN_REVISION="$DEBIAN_REVISION_INPUT" + echo "Using supplied Debian revision: $DEBIAN_REVISION" + else + DEBIAN_REVISION=$(ci/scripts/resolve-matrix.py \ + --type Daily \ + --build "$BUILD_INPUT" \ + --suite "$DISTRO" \ + --field debian_revision) || { + echo "::error::No Daily matrix entry named $BUILD_INPUT for suite $DISTRO." + echo "::error::Add one to ci/build-matrix.yaml, or set the debian-revision input to build this combination ad hoc." + exit 1 + } + echo "Matrix Debian revision for $BUILD_INPUT/$DISTRO: $DEBIAN_REVISION" + fi + echo "DEBIAN_REVISION=$DEBIAN_REVISION" >> "$GITHUB_ENV" + + - name: Merge qcom-next PR patches + if: ${{ inputs.qcom-next-pr != '' }} + shell: bash + env: + QCOM_NEXT_PR: ${{ inputs.qcom-next-pr }} + run: | + cd "$KERNEL_DIR" + echo "Merging qcom-next PR(s): $QCOM_NEXT_PR" + for pr in $QCOM_NEXT_PR; do + [[ "$pr" =~ ^[0-9]+$ ]] || { echo "::error::invalid qcom-next PR number: $pr"; exit 1; } + echo "::group::Merging qcom-next PR #$pr" + git fetch --no-tags origin "pull/$pr/head:pr-$pr" + if ! git merge --no-ff --no-commit "pr-$pr"; then + echo "ERROR: Merge conflict while merging PR #$pr. Aborting." + git merge --abort || true + exit 1 + fi + if ! git diff --cached --quiet; then + git commit -m "Merged qcom-next PR #$pr" + echo "PR #$pr merged successfully." + else + echo "PR #$pr already present; nothing to merge." + fi + echo "::endgroup::" + done + + - name: Apply kernel-topics PR patches + if: ${{ inputs.kernel-topics-pr != '' }} + shell: bash + env: + KERNEL_TOPICS_PR: ${{ inputs.kernel-topics-pr }} + run: | + cd "$KERNEL_DIR" + echo "Applying kernel-topics PR patch(es): $KERNEL_TOPICS_PR" + for pr in $KERNEL_TOPICS_PR; do + [[ "$pr" =~ ^[0-9]+$ ]] || { echo "::error::invalid kernel-topics PR number: $pr"; exit 1; } + echo "::group::Applying kernel-topics PR #$pr" + wget -q "https://github.com/qualcomm-linux/kernel-topics/pull/$pr.patch" -O "$pr.patch" + if ! git am "$pr.patch"; then + echo "ERROR: Patch application failed for PR #$pr. Aborting." + git am --abort || true + exit 1 + fi + echo "PR #$pr applied successfully." + echo "::endgroup::" + done + + - name: Prepare source + shell: bash + env: + SRCPKG_INPUT: ${{ inputs.srcpkg }} + BINPKG_INPUT: ${{ inputs.binpkg }} + KVER_EXTRA_INPUT: ${{ inputs.kver-extra }} + KERNEL_CONFIG_INPUT: ${{ inputs.kernel-config }} + DKMS_INPUT: ${{ inputs.dkms }} + DEBUG_BUILD_INPUT: ${{ inputs.debug-build }} + run: | + # No -x here: keep xtrace off for this step as it assembles the + # prepare-source.sh argument list which may include sensitive paths. + set -euo pipefail + + ARGS=( + --source-dir "$KERNEL_DIR" + --distro "$DISTRO" + --srcpkg "$SRCPKG_INPUT" + --binpkg "$BINPKG_INPUT" + --debian-revision "$DEBIAN_REVISION" + ) + + [[ -n "$LOCALVERSION" ]] && ARGS+=(--localversion "$LOCALVERSION") + [[ -n "$KVER_EXTRA_INPUT" ]] && ARGS+=(--kver-extra "$KVER_EXTRA_INPUT") + [[ -n "$KERNEL_CONFIG_INPUT" ]] && ARGS+=(--kernel-config "$KERNEL_CONFIG_INPUT") + # Always passed, unlike the optional inputs above: an empty list is a + # genuine "bundle nothing" rather than a request for some default. + ARGS+=(--dkms "$DKMS_INPUT") + [[ "$DEBUG_BUILD_INPUT" == "true" ]] && ARGS+=(--debug) + + docker run -i --rm \ + -v "$PWD:$PWD" --workdir="$PWD" \ + "ghcr.io/qualcomm-linux/pkg-builder:$DOCKER_DISTRO" \ + ./prepare-source.sh "${ARGS[@]}" + + - name: Pack prepared source tree + shell: bash + run: | + set -euo pipefail + tar czf /tmp/kernel-srcpkg.tar.gz \ + -C "$(dirname "$KERNEL_DIR")" \ + "$(basename "$KERNEL_DIR")" + + - name: Upload prepared source tree artifact + uses: actions/upload-artifact@v4 + with: + name: kernel-srcpkg-${{ inputs.flavour }}-${{ inputs.suite }} + path: /tmp/kernel-srcpkg.tar.gz + if-no-files-found: error + retention-days: 7 diff --git a/.github/workflows/build-kernel-deb.yml b/.github/workflows/build-kernel-deb.yml deleted file mode 100644 index 184e5896..00000000 --- a/.github/workflows/build-kernel-deb.yml +++ /dev/null @@ -1,431 +0,0 @@ -name: build-kernel-deb - -on: - workflow_dispatch: - inputs: - build: - description: 'Name of this build, as listed in ci/build-matrix.yaml' - required: true - default: 'qcom-next-trixie' - type: string - flavour: - description: 'Kernel flavour: the LOCALVERSION suffix and so the kernel release identity' - required: true - default: 'qcom-next' - type: string - suite: - description: 'Target suite' - required: true - default: 'trixie' - type: string - ref-strategy: - description: 'Kernel ref strategy' - required: true - default: 'latest_tag' - type: choice - options: - - latest_tag - - branch_tip - - pinned_ref - kernel-branch: - description: 'Branch for branch_tip or immutable ref for pinned_ref (ignored for latest_tag)' - required: false - default: 'qcom-next' - tag-pattern: - description: 'Tag glob used only by latest_tag (ignored for branch_tip and pinned_ref)' - required: false - default: 'qcom-next-*' - kernel-url: - description: 'Advanced: custom kernel repository URL (empty uses qualcomm-linux/kernel)' - required: false - default: '' - srcpkg: - description: 'Advanced: Debian source package name' - required: false - default: 'linux-qcom-next' - binpkg: - description: 'Advanced: kernel image metapackage name' - required: false - default: 'linux-image-qcom-next' - kernel-config: - description: 'Advanced: extra config fragments applied on top of all of debian/config-available/ (e.g. intree:arch/arm64/configs/qcom_debug.config)' - required: false - default: '' - dkms: - description: 'Advanced: out-of-tree DKMS modules to build and bundle into the kernel image, comma-separated and without the -dkms suffix (e.g. kgsl,camx); empty bundles nothing' - required: false - default: '' - debian-revision: - description: 'Advanced: Debian revision override (empty takes the Daily revision the matrix gives this variant and suite)' - required: false - default: '' - localversion: - description: 'Advanced: LOCALVERSION override (auto-derived from the resolved ref if empty)' - required: false - default: '' - kver-extra: - description: 'Advanced: extra package-version suffix (e.g. -ci42)' - required: false - default: '' - debug-build: - description: 'Advanced: enable the debug build configuration' - type: boolean - required: false - default: false - qcom-next-pr: - description: 'Advanced Qualcomm-only override: qcom-next PR numbers to merge (e.g. "42 43")' - type: string - required: false - default: '' - kernel-topics-pr: - description: 'Advanced Qualcomm-only override: kernel-topics PR numbers to apply as patches' - type: string - required: false - default: '' - - # Called by daily.yml and release.yml as a reusable workflow. - workflow_call: - inputs: - build: - description: 'Name of this build, as listed in ci/build-matrix.yaml' - type: string - default: 'qcom-next-trixie' - flavour: - description: 'Kernel flavour: the LOCALVERSION suffix and so the kernel release identity' - type: string - default: 'qcom-next' - suite: - description: 'Target suite (trixie, forky, noble, questing, resolute, sid)' - type: string - default: 'trixie' - kernel-branch: - description: 'Source branch or pinned tag used by branch_tip or pinned_ref' - type: string - default: 'qcom-next' - git-clone: - description: 'Kernel repository URL' - type: string - default: 'https://github.com/qualcomm-linux/kernel' - ref-strategy: - description: 'Kernel ref strategy: latest_tag, branch_tip, or pinned_ref' - type: string - default: 'latest_tag' - tag-pattern: - description: 'Tag glob used when ref-strategy=latest_tag' - type: string - default: 'qcom-next-*' - srcpkg: - description: 'Source package name' - type: string - default: 'linux-qcom-next' - binpkg: - description: 'Binary metapackage name' - type: string - default: 'linux-image-qcom-next' - kernel-config: - description: 'Extra config fragments applied on top of all of debian/config-available/ (e.g. intree:arch/arm64/configs/qcom_debug.config)' - type: string - default: '' - dkms: - description: 'Out-of-tree DKMS modules to build and bundle into the kernel image, comma-separated and without the -dkms suffix (e.g. kgsl,camx); empty bundles nothing' - type: string - default: '' - debian-revision: - description: 'Debian revision component of the package version' - type: string - default: '0qli~' - localversion: - description: 'Override LOCALVERSION suffix (auto-derived from the resolved ref if empty)' - type: string - default: '' - kver-extra: - description: 'Extra suffix appended to the package version' - type: string - default: '' - debusine-parent-workspace: - description: 'Parent Debusine workspace for CI child workspace creation' - type: string - default: '' - target-workspace: - description: 'Debusine target workspace for release promotion (empty = daily/S3 path)' - type: string - default: '' - secrets: - DEBUSINE_USER: - description: 'Debusine account used to submit CI and release builds' - required: false - DEBUSINE_TOKEN: - description: 'Debusine token for the CI child workspace' - required: false - DEBUSINE_RELEASE_TOKEN: - description: 'Debusine token for package-publish promotion to the release target workspace' - required: false - -permissions: - contents: read - packages: read - -jobs: - # --------------------------------------------------------------------------- - # prepare: clone kernel source, inject packaging, run prepare-source.sh. - # --------------------------------------------------------------------------- - prepare: - name: Prepare kernel source - runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] - - env: - KERNEL_URL: ${{ inputs.git-clone || inputs.kernel-url || 'https://github.com/qualcomm-linux/kernel' }} - DISTRO: ${{ inputs.suite || 'trixie' }} - # For Debian-family suites (trixie, forky, sid) use trixie as the docker - # image distro: docker-pkg-build and pkg-builder only support trixie for - # Debian. Debusine handles the actual suite-specific build internally. - # For Ubuntu-family suites (noble, questing, resolute) use the real suite. - DOCKER_DISTRO: ${{ (inputs.suite == 'forky' || inputs.suite == 'sid' || inputs.suite == 'unstable') && 'trixie' || inputs.suite || 'trixie' }} - - defaults: - run: - shell: bash - - steps: - - name: Checkout pkg-linux-qcom - # debian/ and ci/ live on the same branch, so one checkout supplies - # both the packaging and resolve-kernel-ref.sh, derive-localversion.sh, - # resolve-matrix.py and the delivery matrix it reads. - # - # An empty ref leaves actions/checkout on github.sha, the commit this - # run was dispatched from or the one the calling workflow runs at. - # Credentials are kept: the self-pr step below fetches from origin. - uses: actions/checkout@v4 - - - name: Checkout docker-pkg-build - uses: actions/checkout@v4 - with: - repository: qualcomm-linux/docker-pkg-build - ref: main - path: docker-pkg-build - - - name: Build docker image - run: | - ./docker-pkg-build/docker_deb_build.py --rebuild -d "$DOCKER_DISTRO" - docker image ls - - - name: Resolve kernel ref - id: kernel-ref - env: - BRANCH_INPUT: ${{ inputs.kernel-branch || 'qcom-next' }} - REF_STRATEGY_INPUT: ${{ inputs.ref-strategy || 'latest_tag' }} - TAG_PATTERN_INPUT: ${{ inputs.tag-pattern || 'qcom-next-*' }} - BUILD_INPUT: ${{ inputs.build || 'qcom-next-trixie' }} - run: | - set -euo pipefail - BRANCH="$BRANCH_INPUT" - REF_STRATEGY="$REF_STRATEGY_INPUT" - TAG_PATTERN="$TAG_PATTERN_INPUT" - - case "$REF_STRATEGY" in - latest_tag) - KERNEL_REF=$(ci/scripts/resolve-kernel-ref.sh \ - --url "$KERNEL_URL" \ - --latest-tag "$TAG_PATTERN") - ;; - branch_tip|pinned_ref) - KERNEL_REF=$(ci/scripts/resolve-kernel-ref.sh --url "$KERNEL_URL" --ref "$BRANCH") - ;; - *) - echo "ERROR: unsupported ref strategy: $REF_STRATEGY" >&2 - exit 1 - ;; - esac - - echo "KERNEL_REF=$KERNEL_REF" >> "$GITHUB_ENV" - echo "Build: $BUILD_INPUT" - echo "Ref strategy: $REF_STRATEGY" - echo "Resolved kernel ref: $KERNEL_REF" - - - name: Clone kernel source - run: | - KERNEL_DIR="$GITHUB_WORKSPACE/kernel-source" - echo "KERNEL_DIR=$KERNEL_DIR" >> "$GITHUB_ENV" - echo "Cloning $KERNEL_URL @ $KERNEL_REF (shallow)..." - git clone --depth 1 --single-branch --branch "$KERNEL_REF" --no-tags \ - "$KERNEL_URL" "$KERNEL_DIR" - KERNEL_SHA=$(git -C "$KERNEL_DIR" rev-parse HEAD) - echo "KERNEL_SHA=$KERNEL_SHA" >> "$GITHUB_ENV" - echo "Kernel HEAD SHA: $KERNEL_SHA" - - - name: Derive LOCALVERSION - env: - LOCALVERSION_INPUT: ${{ inputs.localversion }} - FLAVOUR_INPUT: ${{ inputs.flavour || 'qcom-next' }} - run: | - set -euo pipefail - OVERRIDE_LV="$LOCALVERSION_INPUT" - if [[ -n "$OVERRIDE_LV" ]]; then - echo "Using explicit LOCALVERSION override: $OVERRIDE_LV" - echo "LOCALVERSION=$OVERRIDE_LV" >> "$GITHUB_ENV" - else - LOCALVERSION=$(ci/scripts/derive-localversion.sh \ - --flavour "$FLAVOUR_INPUT" \ - --ref "$KERNEL_REF" \ - --sha "$(echo "$KERNEL_SHA" | cut -c1-12)") - echo "LOCALVERSION=$LOCALVERSION" >> "$GITHUB_ENV" - echo "Derived LOCALVERSION: $LOCALVERSION" - fi - - - name: Resolve Debian revision - # workflow_call callers (daily.yml, release.yml) pass the entry's own - # debian-revision straight from the matrix. A direct workflow_dispatch - # that leaves it empty looks the same entry up itself, so a manual - # build of a configured variant and suite gets exactly the revision the - # daily build would have used. The lookup is always the Daily entry: - # direct dispatch is build-only and never promotes. - env: - DEBIAN_REVISION_INPUT: ${{ inputs.debian-revision }} - BUILD_INPUT: ${{ inputs.build || 'qcom-next-trixie' }} - run: | - set -euo pipefail - [[ "$BUILD_INPUT" =~ ^[a-z0-9]+([a-z0-9-]*[a-z0-9])?$ ]] || { - echo "::error::build name must use lowercase letters, digits, and internal hyphens" - exit 1 - } - INPUT_REVISION="$DEBIAN_REVISION_INPUT" - if [[ -n "$INPUT_REVISION" ]]; then - DEBIAN_REVISION="$INPUT_REVISION" - echo "Using supplied Debian revision: $DEBIAN_REVISION" - else - DEBIAN_REVISION=$(ci/scripts/resolve-matrix.py \ - --type Daily \ - --build "$BUILD_INPUT" \ - --suite "$DISTRO" \ - --field debian_revision) || { - echo "::error::No Daily matrix entry named $BUILD_INPUT for suite $DISTRO." - echo "::error::Add one to ci/build-matrix.yaml, or set the debian-revision input to build this combination ad hoc." - exit 1 - } - echo "Matrix Debian revision for $BUILD_INPUT/$DISTRO: $DEBIAN_REVISION" - fi - echo "DEBIAN_REVISION=$DEBIAN_REVISION" >> "$GITHUB_ENV" - - - name: Merge qcom-next PR patches - if: ${{ inputs.qcom-next-pr != '' }} - env: - QCOM_NEXT_PR: ${{ inputs.qcom-next-pr }} - run: | - cd "$KERNEL_DIR" - echo "Merging qcom-next PR(s): $QCOM_NEXT_PR" - for pr in $QCOM_NEXT_PR; do - [[ "$pr" =~ ^[0-9]+$ ]] || { echo "::error::invalid qcom-next PR number: $pr"; exit 1; } - echo "::group::Merging qcom-next PR #$pr" - git fetch --no-tags origin "pull/$pr/head:pr-$pr" - if ! git merge --no-ff --no-commit "pr-$pr"; then - echo "ERROR: Merge conflict while merging PR #$pr. Aborting." - git merge --abort || true - exit 1 - fi - if ! git diff --cached --quiet; then - git commit -m "Merged qcom-next PR #$pr" - echo "PR #$pr merged successfully." - else - echo "PR #$pr already present; nothing to merge." - fi - echo "::endgroup::" - done - - - name: Apply kernel-topics PR patches - if: ${{ inputs.kernel-topics-pr != '' }} - env: - KERNEL_TOPICS_PR: ${{ inputs.kernel-topics-pr }} - run: | - cd "$KERNEL_DIR" - echo "Applying kernel-topics PR patch(es): $KERNEL_TOPICS_PR" - for pr in $KERNEL_TOPICS_PR; do - [[ "$pr" =~ ^[0-9]+$ ]] || { echo "::error::invalid kernel-topics PR number: $pr"; exit 1; } - echo "::group::Applying kernel-topics PR #$pr" - wget -q "https://github.com/qualcomm-linux/kernel-topics/pull/$pr.patch" -O "$pr.patch" - if ! git am "$pr.patch"; then - echo "ERROR: Patch application failed for PR #$pr. Aborting." - git am --abort || true - exit 1 - fi - echo "PR #$pr applied successfully." - echo "::endgroup::" - done - - - name: Prepare source - env: - SRCPKG_INPUT: ${{ inputs.srcpkg || 'linux-qcom-next' }} - BINPKG_INPUT: ${{ inputs.binpkg || 'linux-image-qcom-next' }} - KVER_EXTRA_INPUT: ${{ inputs.kver-extra }} - KERNEL_CONFIG_INPUT: ${{ inputs.kernel-config }} - DKMS_INPUT: ${{ inputs.dkms }} - DEBUG_BUILD_INPUT: ${{ inputs.debug-build }} - run: | - # No -x here: keep xtrace off for this step as it assembles the - # prepare-source.sh argument list which may include sensitive paths. - set -euo pipefail - - ARGS=( - --source-dir "$KERNEL_DIR" - --distro "$DISTRO" - --srcpkg "$SRCPKG_INPUT" - --binpkg "$BINPKG_INPUT" - --debian-revision "$DEBIAN_REVISION" - ) - - [[ -n "$LOCALVERSION" ]] && ARGS+=(--localversion "$LOCALVERSION") - [[ -n "$KVER_EXTRA_INPUT" ]] && ARGS+=(--kver-extra "$KVER_EXTRA_INPUT") - [[ -n "$KERNEL_CONFIG_INPUT" ]] && ARGS+=(--kernel-config "$KERNEL_CONFIG_INPUT") - # Always passed, unlike the optional inputs above: an empty list is a - # genuine "bundle nothing" rather than a request for some default. - ARGS+=(--dkms "$DKMS_INPUT") - [[ "$DEBUG_BUILD_INPUT" == "true" ]] && ARGS+=(--debug) - - docker run -i --rm \ - -v "$PWD:$PWD" --workdir="$PWD" \ - "ghcr.io/qualcomm-linux/pkg-builder:$DOCKER_DISTRO" \ - ./prepare-source.sh "${ARGS[@]}" - - - name: Upload prepared source tree - run: | - tar czf /tmp/kernel-srcpkg.tar.gz \ - -C "$(dirname "$KERNEL_DIR")" \ - "$(basename "$KERNEL_DIR")" - - - name: Upload prepared source tree artifact - uses: actions/upload-artifact@v4 - with: - name: kernel-srcpkg-${{ inputs.flavour }}-${{ inputs.suite }} - path: /tmp/kernel-srcpkg.tar.gz - if-no-files-found: error - retention-days: 7 - - # --------------------------------------------------------------------------- - # debusine-build: Debian suites only (trixie, forky, sid). - # --------------------------------------------------------------------------- - debusine-build: - name: Build in Debusine - needs: [prepare] - if: ${{ contains(fromJSON('["trixie", "forky", "sid", "unstable", "bookworm"]'), inputs.suite || 'trixie') }} - uses: ./.github/workflows/build-kernel-debusine.yml - with: - suite: ${{ inputs.suite || 'trixie' }} - flavour: ${{ inputs.flavour || 'qcom-next' }} - srcpkg: ${{ inputs.srcpkg || 'linux-qcom-next' }} - debusine-parent-workspace: ${{ inputs.debusine-parent-workspace || vars.DEBUSINE_PARENT_WORKSPACE || 'qli-ci' }} - target-workspace: ${{ inputs.target-workspace || '' }} - secrets: - DEBUSINE_USER: ${{ secrets.DEBUSINE_USER }} - DEBUSINE_TOKEN: ${{ secrets.DEBUSINE_TOKEN }} - DEBUSINE_RELEASE_TOKEN: ${{ secrets.DEBUSINE_RELEASE_TOKEN }} - - # --------------------------------------------------------------------------- - # ubuntu-build: Ubuntu suites only (noble, questing, resolute). - # --------------------------------------------------------------------------- - ubuntu-build: - name: Build (Ubuntu, docker) - needs: [prepare] - if: ${{ !contains(fromJSON('["trixie", "forky", "sid", "unstable", "bookworm"]'), inputs.suite || 'trixie') }} - uses: ./.github/workflows/build-kernel-ubuntu.yml - with: - distro: ${{ inputs.suite || 'resolute' }} - flavour: ${{ inputs.flavour || 'qcom-next' }} diff --git a/.github/workflows/build-kernel-debian.yml b/.github/workflows/build-kernel-debian.yml new file mode 100644 index 00000000..a885bcea --- /dev/null +++ b/.github/workflows/build-kernel-debian.yml @@ -0,0 +1,328 @@ +name: build-kernel-debian + +# Build a kernel package for a Debian-family suite and publish it to S3. +# +# Called per matrix entry by daily.yml and pr-build.yml, which select the +# Debian entries with resolve-matrix.py --family debian and call only this +# workflow for them. Nothing here is conditional on the suite: a caller that +# reaches this file has already decided the build belongs on the Debusine +# path, so every job in it runs. +# +# release-kernel-debian.yml is the same build with the release tail: it +# promotes to a target Debusine workspace instead of publishing to S3. +# +# Jobs: +# prepare (self-hosted): clone, patch, inject debian/, generate the source. +# build (debusine-pkg-builder container): submit to Debusine. +# publish (self-hosted): download the .deb files and upload them to S3. + +on: + workflow_call: + inputs: + build: + description: 'Name of this build, as listed in ci/build-matrix.yaml' + type: string + default: 'qcom-next-trixie' + flavour: + description: 'Kernel flavour: the LOCALVERSION suffix and so the kernel release identity' + type: string + default: 'qcom-next' + suite: + description: 'Target Debian suite (trixie, forky, sid)' + type: string + default: 'trixie' + kernel-branch: + description: 'Source branch or pinned tag used by branch_tip or pinned_ref' + type: string + default: 'qcom-next' + git-clone: + description: 'Kernel repository URL' + type: string + default: 'https://github.com/qualcomm-linux/kernel' + ref-strategy: + description: 'Kernel ref strategy: latest_tag, branch_tip, or pinned_ref' + type: string + default: 'latest_tag' + tag-pattern: + description: 'Tag glob used when ref-strategy=latest_tag' + type: string + default: 'qcom-next-*' + srcpkg: + description: 'Source package name' + type: string + default: 'linux-qcom-next' + binpkg: + description: 'Binary metapackage name' + type: string + default: 'linux-image-qcom-next' + kernel-config: + description: 'Extra config fragments applied on top of all of debian/config-available/' + type: string + default: '' + dkms: + description: 'Out-of-tree DKMS modules to bundle, comma-separated and without the -dkms suffix' + type: string + default: '' + debian-revision: + description: 'Debian revision component of the package version' + type: string + default: '0qli~' + localversion: + description: 'Override LOCALVERSION suffix (auto-derived from the resolved ref if empty)' + type: string + default: '' + kver-extra: + description: 'Extra suffix appended to the package version' + type: string + default: '' + debug-build: + description: 'Enable the debug build configuration' + type: boolean + default: false + debusine-parent-workspace: + description: 'Parent Debusine workspace for CI child workspace creation' + type: string + default: '' + qcom-next-pr: + description: 'qcom-next PR numbers to merge (e.g. "42 43")' + type: string + default: '' + kernel-topics-pr: + description: 'kernel-topics PR numbers to apply as patches' + type: string + default: '' + secrets: + DEBUSINE_USER: + description: 'Debusine account used to submit CI builds' + required: false + DEBUSINE_TOKEN: + description: 'Debusine token for the CI child workspace' + required: false + + # Ad-hoc build of one Debian suite. Release promotion is not offered here: + # a manual build never promotes, so it always takes the S3 publish path. + workflow_dispatch: + inputs: + build: + description: 'Name of this build, as listed in ci/build-matrix.yaml' + required: true + default: 'qcom-next-trixie' + type: string + flavour: + description: 'Kernel flavour: the LOCALVERSION suffix and so the kernel release identity' + required: true + default: 'qcom-next' + type: string + suite: + description: 'Target Debian suite (trixie, forky, sid). An Ubuntu suite here is rejected before anything is built; dispatch build-kernel-ubuntu for those.' + required: true + default: 'trixie' + type: string + ref-strategy: + description: 'Kernel ref strategy' + required: true + default: 'latest_tag' + type: choice + options: + - latest_tag + - branch_tip + - pinned_ref + kernel-branch: + description: 'Branch for branch_tip or immutable ref for pinned_ref (ignored for latest_tag)' + required: false + default: 'qcom-next' + tag-pattern: + description: 'Tag glob used only by latest_tag (ignored for branch_tip and pinned_ref)' + required: false + default: 'qcom-next-*' + git-clone: + description: 'Advanced: custom kernel repository URL' + required: false + default: 'https://github.com/qualcomm-linux/kernel' + srcpkg: + description: 'Advanced: Debian source package name' + required: false + default: 'linux-qcom-next' + binpkg: + description: 'Advanced: kernel image metapackage name' + required: false + default: 'linux-image-qcom-next' + kernel-config: + description: 'Advanced: extra config fragments applied on top of all of debian/config-available/ (e.g. intree:arch/arm64/configs/qcom_debug.config)' + required: false + default: '' + dkms: + description: 'Advanced: out-of-tree DKMS modules to build and bundle into the kernel image, comma-separated and without the -dkms suffix (e.g. kgsl,camx); empty bundles nothing' + required: false + default: '' + debian-revision: + description: 'Advanced: Debian revision override (empty takes the Daily revision the matrix gives this build and suite)' + required: false + default: '' + localversion: + description: 'Advanced: LOCALVERSION override (auto-derived from the resolved ref if empty)' + required: false + default: '' + kver-extra: + description: 'Advanced: extra package-version suffix (e.g. -ci42)' + required: false + default: '' + debug-build: + description: 'Advanced: enable the debug build configuration' + type: boolean + required: false + default: false + qcom-next-pr: + description: 'Advanced Qualcomm-only override: qcom-next PR numbers to merge (e.g. "42 43")' + type: string + required: false + default: '' + kernel-topics-pr: + description: 'Advanced Qualcomm-only override: kernel-topics PR numbers to apply as patches' + type: string + required: false + default: '' + +permissions: + contents: read + packages: read + +jobs: + # --------------------------------------------------------------------------- + # prepare: clone kernel source, inject packaging, run prepare-source.sh. + # --------------------------------------------------------------------------- + prepare: + name: Prepare kernel source + runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] + steps: + # debian/ and ci/ live on the same branch, so one checkout supplies both + # the packaging and the scripts, the delivery matrix they read, and the + # prepare-kernel-source action below. + # + # An empty ref leaves actions/checkout on github.sha, the commit this run + # was dispatched from or the one the calling workflow runs at. + - name: Checkout pkg-linux-qcom + uses: actions/checkout@v4 + + - name: Prepare kernel source + uses: ./.github/actions/prepare-kernel-source + with: + build: ${{ inputs.build }} + suite: ${{ inputs.suite }} + expect-family: debian + flavour: ${{ inputs.flavour }} + kernel-url: ${{ inputs.git-clone }} + ref-strategy: ${{ inputs.ref-strategy }} + kernel-branch: ${{ inputs.kernel-branch }} + tag-pattern: ${{ inputs.tag-pattern }} + srcpkg: ${{ inputs.srcpkg }} + binpkg: ${{ inputs.binpkg }} + kernel-config: ${{ inputs.kernel-config }} + dkms: ${{ inputs.dkms }} + debian-revision: ${{ inputs.debian-revision }} + localversion: ${{ inputs.localversion }} + kver-extra: ${{ inputs.kver-extra }} + debug-build: ${{ inputs.debug-build }} + qcom-next-pr: ${{ inputs.qcom-next-pr }} + kernel-topics-pr: ${{ inputs.kernel-topics-pr }} + + # --------------------------------------------------------------------------- + # build: generate the Debian source package and submit it to Debusine. + # --------------------------------------------------------------------------- + build: + name: Build (Debusine) + needs: prepare + runs-on: ubuntu-latest + environment: Staging + container: + image: ghcr.io/qualcomm-linux/debusine-pkg-builder:trixie + options: --user 0:0 + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + outputs: + workspace: ${{ steps.debusine.outputs.workspace }} + steps: + # Only for the debusine-build action itself: the package being built + # comes from the prepared-source artifact, not from this checkout. + - name: Checkout pkg-linux-qcom + uses: actions/checkout@v4 + + - name: Build in Debusine + id: debusine + uses: ./.github/actions/debusine-build + with: + suite: ${{ inputs.suite }} + flavour: ${{ inputs.flavour }} + debusine-host: ${{ vars.DEBUSINE_HOST }} + debusine-scope: ${{ vars.DEBUSINE_SCOPE }} + debusine-user: ${{ secrets.DEBUSINE_USER }} + debusine-token: ${{ secrets.DEBUSINE_TOKEN }} + debusine-parent-workspace: ${{ inputs.debusine-parent-workspace || vars.DEBUSINE_PARENT_WORKSPACE || 'qli-ci' }} + # A staging build: also reads qli-staging, which carries package + # versions that have not been released to qli yet. + extra-build-dep-workspaces: qli qli-staging + + # --------------------------------------------------------------------------- + # publish: download the .deb files from Debusine and upload them to S3. + # Runs on the self-hosted runner, which has direct IAM access. + # --------------------------------------------------------------------------- + publish: + name: Publish to S3 + needs: build + runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] + defaults: + run: + shell: bash + steps: + - name: Checkout debusine-action helpers + uses: actions/checkout@v5 + with: + repository: qualcomm-linux/debusine-action + ref: main + path: debusine-action + fetch-depth: 1 + sparse-checkout: | + lib + + - name: Generate Debusine apt configuration + env: + DEBUSINE_HOST: ${{ vars.DEBUSINE_HOST }} + DEBUSINE_SCOPE: ${{ vars.DEBUSINE_SCOPE }} + DEBUSINE_USER: ${{ secrets.DEBUSINE_USER }} + DEBUSINE_TOKEN: ${{ secrets.DEBUSINE_TOKEN }} + DEBUSINE_WORKSPACE: ${{ needs.build.outputs.workspace }} + SUITE: ${{ inputs.suite }} + run: | + debusine-action/lib/generate-apt-config + + - name: Download .deb packages from Debusine workspace + run: | + set -euxo pipefail + sudo apt-get install -y --no-install-recommends devscripts + mkdir -p "$GITHUB_WORKSPACE/deb-artifacts" + chdist create debusine-workspace + rm -f ~/.chdist/debusine-workspace/etc/apt/sources.list + install -d ~/.chdist/debusine-workspace/etc/apt/sources.list.d + install -d ~/.chdist/debusine-workspace/etc/apt/auth.conf.d + install -m 0644 debusine-ci.sources \ + ~/.chdist/debusine-workspace/etc/apt/sources.list.d/ + install -m 0600 debusine-ci-auth.conf \ + ~/.chdist/debusine-workspace/etc/apt/auth.conf.d/ + chdist apt-get debusine-workspace update + packages=$(chdist apt-cache debusine-workspace search . | awk '{print $1}' | tr '\n' ' ') + [[ -n "$packages" ]] || { echo "ERROR: no packages found in Debusine workspace"; exit 1; } + echo "Packages to download: $packages" + cd "$GITHUB_WORKSPACE/deb-artifacts" + # shellcheck disable=SC2086 + chdist apt-get debusine-workspace download $packages + + - name: Upload .deb packages to S3 + # Keep flavour and suite in the destination so concurrent matrix legs + # cannot overwrite or mix package outputs. Consumers must select the + # flavour and suite they intend to install. + uses: qualcomm-linux/upload-private-artifact-action@aws-v4 + with: + s3_bucket: ${{ vars.ARTIFACT_S3_BUCKET }} + path: deb-artifacts + destination: ${{ github.repository_owner }}/pkg/debusine/${{ github.event.repository.name }}/${{ inputs.flavour }}/${{ inputs.suite }}/${{ github.run_id }}-${{ github.run_attempt }}/ diff --git a/.github/workflows/build-kernel-debusine.yml b/.github/workflows/build-kernel-debusine.yml deleted file mode 100644 index 2f21529b..00000000 --- a/.github/workflows/build-kernel-debusine.yml +++ /dev/null @@ -1,242 +0,0 @@ -name: build-debusine - -# Reusable workflow: Debian kernel build via Debusine, then publish. -# -# Called by build-kernel-deb.yml for Debian-family suites (trixie, forky, sid). -# The kernel source is already prepared by the caller's prepare job and shared -# as the kernel-srcpkg artifact. Artifact and workspace identity include both -# flavour and suite so parallel flavours cannot share inputs or outputs. -# -# Publish path is determined by target-workspace: -# target-workspace == '' -> daily path: download .deb files, upload to S3. -# target-workspace != '' -> release path: promote from CI workspace to the -# target Debusine workspace via debusine-action lib/release. -# -# Jobs: -# build (debusine-pkg-builder container): generates .dsc, submits to Debusine. -# publish (self-hosted runner): S3 upload (daily path only). -# release (debusine-pkg-builder container): Debusine promotion (release path only). - -on: - workflow_call: - inputs: - suite: - description: 'Target Debian suite (trixie, forky, sid)' - type: string - required: true - flavour: - description: 'Kernel flavour, isolating artifacts, Debusine workspaces and S3 paths' - type: string - required: true - srcpkg: - description: 'Source package name (e.g. linux-qcom-next)' - type: string - default: 'linux-qcom-next' - debusine-parent-workspace: - description: 'Parent Debusine workspace for CI child workspace creation' - type: string - default: 'qli-ci' - target-workspace: - description: 'Debusine target workspace for release promotion (empty = daily/S3 path)' - type: string - default: '' - secrets: - DEBUSINE_USER: - required: true - DEBUSINE_TOKEN: - required: true - DEBUSINE_RELEASE_TOKEN: - required: false - -permissions: - contents: read - packages: read - -env: - DEBUSINE_ACTION_REF: main - -jobs: - # --------------------------------------------------------------------------- - # build: generate Debian source package and submit to Debusine. - # --------------------------------------------------------------------------- - build: - name: Build (Debusine) - runs-on: ubuntu-latest - container: - image: ghcr.io/qualcomm-linux/debusine-pkg-builder:trixie - options: --user 0:0 - credentials: - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - outputs: - workspace: ${{ steps.build-debusine.outputs.workspace }} - workspace_url: ${{ steps.build-debusine.outputs.workspace_url }} - srcpkg_version: ${{ steps.generate-srcpkg.outputs.srcpkg_version }} - defaults: - run: - shell: bash - environment: ${{ inputs.target-workspace != '' && 'Production' || 'Staging' }} - steps: - - name: Checkout debusine-action helpers - uses: actions/checkout@v5 - with: - repository: qualcomm-linux/debusine-action - ref: ${{ env.DEBUSINE_ACTION_REF }} - path: debusine-action - fetch-depth: 1 - sparse-checkout: | - lib - - - name: Download prepared source tree - uses: actions/download-artifact@v4 - with: - name: kernel-srcpkg-${{ inputs.flavour }}-${{ inputs.suite }} - path: /tmp/srcpkg-artifact - - - name: Extract prepared source tree - run: | - mkdir srcpkg - tar xzf /tmp/srcpkg-artifact/*.tar.gz \ - -C srcpkg --strip-components=1 - - - name: Generate source package - id: generate-srcpkg - env: - SUITE: ${{ inputs.suite }} - DEBUSINE_ASSEMBLE_ORIG: "true" - run: | - SUITE="$SUITE" debusine-action/lib/generate-source-package - - - name: Build in Debusine - id: build-debusine - env: - GITHUB_REPOSITORY_ID: ${{ github.repository_id }} - GITHUB_RUN_ID: ${{ github.run_id }} - GITHUB_RUN_ATTEMPT: ${{ github.run_attempt }} - JOB_INDEX: ${{ inputs.flavour }}-${{ inputs.suite }} - DEBUSINE_HOST: ${{ vars.DEBUSINE_HOST }} - DEBUSINE_SCOPE: ${{ vars.DEBUSINE_SCOPE }} - DEBUSINE_USER: ${{ secrets.DEBUSINE_USER }} - DEBUSINE_TOKEN: ${{ secrets.DEBUSINE_TOKEN }} - DEBUSINE_PARENT_WORKSPACE: ${{ inputs.debusine-parent-workspace }} - SUITE: ${{ inputs.suite }} - EXTRA_BUILD_DEP_WORKSPACES: qli - run: | - # No -x here: DEBUSINE_TOKEN is in env, keep xtrace off. - set -euo pipefail - debusine-action/lib/build - workspace=$(sed -n 's/^workspace=//p' "$GITHUB_OUTPUT") - echo "workspace_url=https://${DEBUSINE_HOST}/${DEBUSINE_SCOPE}/${workspace}/" >> "$GITHUB_OUTPUT" - - - name: Note Debusine workspace URL - env: - WORKSPACE_URL: ${{ steps.build-debusine.outputs.workspace_url }} - run: | - echo "Debusine Workspace URL: $WORKSPACE_URL" >> "$GITHUB_STEP_SUMMARY" - - # --------------------------------------------------------------------------- - # publish: download .deb files from Debusine and upload to S3. - # Daily path only (target-workspace is empty). - # --------------------------------------------------------------------------- - publish: - name: Publish to S3 - needs: build - if: ${{ needs.build.result == 'success' && inputs.target-workspace == '' }} - runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] - defaults: - run: - shell: bash - steps: - - name: Checkout debusine-action helpers - uses: actions/checkout@v5 - with: - repository: qualcomm-linux/debusine-action - ref: ${{ env.DEBUSINE_ACTION_REF }} - path: debusine-action - fetch-depth: 1 - sparse-checkout: | - lib - - - name: Generate Debusine apt configuration - env: - DEBUSINE_HOST: ${{ vars.DEBUSINE_HOST }} - DEBUSINE_SCOPE: ${{ vars.DEBUSINE_SCOPE }} - DEBUSINE_USER: ${{ secrets.DEBUSINE_USER }} - DEBUSINE_TOKEN: ${{ secrets.DEBUSINE_TOKEN }} - DEBUSINE_WORKSPACE: ${{ needs.build.outputs.workspace }} - SUITE: ${{ inputs.suite }} - run: | - debusine-action/lib/generate-apt-config - - - name: Download .deb packages from Debusine workspace - run: | - set -euxo pipefail - sudo apt-get install -y --no-install-recommends devscripts - mkdir -p "$GITHUB_WORKSPACE/deb-artifacts" - chdist create debusine-workspace - rm -f ~/.chdist/debusine-workspace/etc/apt/sources.list - install -d ~/.chdist/debusine-workspace/etc/apt/sources.list.d - install -d ~/.chdist/debusine-workspace/etc/apt/auth.conf.d - install -m 0644 debusine-ci.sources \ - ~/.chdist/debusine-workspace/etc/apt/sources.list.d/ - install -m 0600 debusine-ci-auth.conf \ - ~/.chdist/debusine-workspace/etc/apt/auth.conf.d/ - chdist apt-get debusine-workspace update - packages=$(chdist apt-cache debusine-workspace search . | awk '{print $1}' | tr '\n' ' ') - [[ -n "$packages" ]] || { echo "ERROR: no packages found in Debusine workspace"; exit 1; } - echo "Packages to download: $packages" - cd "$GITHUB_WORKSPACE/deb-artifacts" - # shellcheck disable=SC2086 - chdist apt-get debusine-workspace download $packages - - - name: Upload .deb packages to S3 - uses: qualcomm-linux/upload-private-artifact-action@aws-v4 - with: - s3_bucket: ${{ vars.ARTIFACT_S3_BUCKET }} - path: deb-artifacts - destination: ${{ github.repository_owner }}/pkg/debusine/${{ github.event.repository.name }}/${{ inputs.flavour }}/${{ inputs.suite }}/${{ github.run_id }}-${{ github.run_attempt }}/ - - # --------------------------------------------------------------------------- - # release: promote packages from CI workspace to target Debusine workspace. - # Release path only (target-workspace is non-empty). - # --------------------------------------------------------------------------- - release: - name: Release to Debusine - needs: build - if: ${{ needs.build.result == 'success' && inputs.target-workspace != '' }} - runs-on: ubuntu-latest - environment: Production - container: - image: ghcr.io/qualcomm-linux/debusine-pkg-builder:trixie - options: --user 0:0 - credentials: - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - defaults: - run: - shell: bash - steps: - - name: Checkout debusine-action helpers - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5 - with: - persist-credentials: false - repository: qualcomm-linux/debusine-action - ref: ${{ env.DEBUSINE_ACTION_REF }} - path: debusine-action - fetch-depth: 1 - sparse-checkout: | - lib - - - name: Promote packages to target workspace - env: - DEBUSINE_HOST: ${{ vars.DEBUSINE_HOST }} - DEBUSINE_SCOPE: ${{ vars.DEBUSINE_SCOPE }} - DEBUSINE_TOKEN: ${{ secrets.DEBUSINE_RELEASE_TOKEN }} - DEBUSINE_CI_WORKSPACE: ${{ needs.build.outputs.workspace }} - DEBUSINE_TARGET_WORKSPACE: ${{ inputs.target-workspace }} - SRCPKG_NAME: ${{ inputs.srcpkg }} - SRCPKG_VERSION: ${{ needs.build.outputs.srcpkg_version }} - SUITE: ${{ inputs.suite }} - run: | - set -ex - debusine-action/lib/release diff --git a/.github/workflows/build-kernel-ubuntu.yml b/.github/workflows/build-kernel-ubuntu.yml index 97a15264..1e4a8aa1 100644 --- a/.github/workflows/build-kernel-ubuntu.yml +++ b/.github/workflows/build-kernel-ubuntu.yml @@ -1,43 +1,225 @@ name: build-kernel-ubuntu -# Reusable workflow: Ubuntu kernel build via build-kernel.sh. +# Build a kernel package for an Ubuntu-family suite and publish it to S3. # -# Called by build-kernel-deb.yml when the target suite is Ubuntu-family -# (noble, questing, resolute). The kernel source has already been cloned, -# patched, and prepared (debian/ injected, debian/control + debian/changelog -# generated) by the prepare job in the caller workflow. This workflow -# downloads the kernel-srcpkg artifact and runs build-kernel.sh --skip-prepare -# to go directly to the build step. +# Called per matrix entry by daily.yml and pr-build.yml, which select the +# Ubuntu entries with resolve-matrix.py --family ubuntu and call only this +# workflow for them. The package is built in a suite-matched pkg-builder +# container on the self-hosted runner rather than in Debusine, which builds +# only the Debian family. # -# The artifact is shared from the caller via github.run_id. Build and S3 upload -# run on the self-hosted runner, which has direct IAM access. +# Ubuntu has no release path: promotion runs through Debusine, so a Release +# entry for an Ubuntu suite is rejected by resolve-matrix.py rather than built +# here and quietly not promoted. +# +# Jobs: +# prepare (self-hosted): clone, patch, inject debian/, generate the source. +# build (self-hosted): build the .deb files and upload them to S3. on: workflow_call: inputs: - distro: - description: Target Ubuntu suite (noble, questing, resolute) + build: + description: 'Name of this build, as listed in ci/build-matrix.yaml' + type: string + default: 'qcom-next-resolute' + flavour: + description: 'Kernel flavour: the LOCALVERSION suffix and so the kernel release identity' + type: string + default: 'qcom-next' + suite: + description: 'Target Ubuntu suite (noble, questing, resolute)' + type: string + default: 'resolute' + kernel-branch: + description: 'Source branch or pinned tag used by branch_tip or pinned_ref' + type: string + default: 'qcom-next' + git-clone: + description: 'Kernel repository URL' + type: string + default: 'https://github.com/qualcomm-linux/kernel' + ref-strategy: + description: 'Kernel ref strategy: latest_tag, branch_tip, or pinned_ref' + type: string + default: 'latest_tag' + tag-pattern: + description: 'Tag glob used when ref-strategy=latest_tag' + type: string + default: 'qcom-next-*' + srcpkg: + description: 'Source package name' + type: string + default: 'linux-qcom-next' + binpkg: + description: 'Binary metapackage name' + type: string + default: 'linux-image-qcom-next' + kernel-config: + description: 'Extra config fragments applied on top of all of debian/config-available/' + type: string + default: '' + dkms: + description: 'Out-of-tree DKMS modules to bundle, comma-separated and without the -dkms suffix' + type: string + default: '' + debian-revision: + description: 'Debian revision component of the package version' + type: string + default: '0qli~' + localversion: + description: 'Override LOCALVERSION suffix (auto-derived from the resolved ref if empty)' + type: string + default: '' + kver-extra: + description: 'Extra suffix appended to the package version' + type: string + default: '' + debug-build: + description: 'Enable the debug build configuration' + type: boolean + default: false + qcom-next-pr: + description: 'qcom-next PR numbers to merge (e.g. "42 43")' + type: string + default: '' + kernel-topics-pr: + description: 'kernel-topics PR numbers to apply as patches' type: string + default: '' + + # Ad-hoc build of one Ubuntu suite. + workflow_dispatch: + inputs: + build: + description: 'Name of this build, as listed in ci/build-matrix.yaml' required: true + default: 'qcom-next-resolute' + type: string flavour: - description: 'Kernel flavour, isolating prepared-source artifacts and published packages' + description: 'Kernel flavour: the LOCALVERSION suffix and so the kernel release identity' + required: true + default: 'qcom-next' type: string + suite: + description: 'Target Ubuntu suite (noble, questing, resolute). A Debian suite here is rejected before anything is built; dispatch build-kernel-debian for those.' required: true + default: 'resolute' + type: string + ref-strategy: + description: 'Kernel ref strategy' + required: true + default: 'latest_tag' + type: choice + options: + - latest_tag + - branch_tip + - pinned_ref + kernel-branch: + description: 'Branch for branch_tip or immutable ref for pinned_ref (ignored for latest_tag)' + required: false + default: 'qcom-next' + tag-pattern: + description: 'Tag glob used only by latest_tag (ignored for branch_tip and pinned_ref)' + required: false + default: 'qcom-next-*' + git-clone: + description: 'Advanced: custom kernel repository URL' + required: false + default: 'https://github.com/qualcomm-linux/kernel' + srcpkg: + description: 'Advanced: Debian source package name' + required: false + default: 'linux-qcom-next' + binpkg: + description: 'Advanced: kernel image metapackage name' + required: false + default: 'linux-image-qcom-next' + kernel-config: + description: 'Advanced: extra config fragments applied on top of all of debian/config-available/ (e.g. intree:arch/arm64/configs/qcom_debug.config)' + required: false + default: '' + dkms: + description: 'Advanced: out-of-tree DKMS modules to build and bundle into the kernel image, comma-separated and without the -dkms suffix (e.g. kgsl,camx); empty bundles nothing' + required: false + default: '' + debian-revision: + description: 'Advanced: Debian revision override (empty takes the Daily revision the matrix gives this build and suite)' + required: false + default: '' + localversion: + description: 'Advanced: LOCALVERSION override (auto-derived from the resolved ref if empty)' + required: false + default: '' + kver-extra: + description: 'Advanced: extra package-version suffix (e.g. -ci42)' + required: false + default: '' + debug-build: + description: 'Advanced: enable the debug build configuration' + type: boolean + required: false + default: false + qcom-next-pr: + description: 'Advanced Qualcomm-only override: qcom-next PR numbers to merge (e.g. "42 43")' + type: string + required: false + default: '' + kernel-topics-pr: + description: 'Advanced Qualcomm-only override: kernel-topics PR numbers to apply as patches' + type: string + required: false + default: '' permissions: contents: read packages: read jobs: + # --------------------------------------------------------------------------- + # prepare: clone kernel source, inject packaging, run prepare-source.sh. + # --------------------------------------------------------------------------- + prepare: + name: Prepare kernel source + runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] + steps: + - name: Checkout pkg-linux-qcom + uses: actions/checkout@v4 + + - name: Prepare kernel source + uses: ./.github/actions/prepare-kernel-source + with: + build: ${{ inputs.build }} + suite: ${{ inputs.suite }} + expect-family: ubuntu + flavour: ${{ inputs.flavour }} + kernel-url: ${{ inputs.git-clone }} + ref-strategy: ${{ inputs.ref-strategy }} + kernel-branch: ${{ inputs.kernel-branch }} + tag-pattern: ${{ inputs.tag-pattern }} + srcpkg: ${{ inputs.srcpkg }} + binpkg: ${{ inputs.binpkg }} + kernel-config: ${{ inputs.kernel-config }} + dkms: ${{ inputs.dkms }} + debian-revision: ${{ inputs.debian-revision }} + localversion: ${{ inputs.localversion }} + kver-extra: ${{ inputs.kver-extra }} + debug-build: ${{ inputs.debug-build }} + qcom-next-pr: ${{ inputs.qcom-next-pr }} + kernel-topics-pr: ${{ inputs.kernel-topics-pr }} + + # --------------------------------------------------------------------------- + # build: build the prepared source in a suite-matched container, then publish. + # --------------------------------------------------------------------------- build: name: Build kernel package + needs: prepare runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] env: - DISTRO: ${{ inputs.distro }} + DISTRO: ${{ inputs.suite }} defaults: run: shell: bash - steps: - name: Checkout pkg-linux-qcom uses: actions/checkout@v4 @@ -60,7 +242,7 @@ jobs: - name: Download prepared source tree uses: actions/download-artifact@v4 with: - name: kernel-srcpkg-${{ inputs.flavour }}-${{ inputs.distro }} + name: kernel-srcpkg-${{ inputs.flavour }}-${{ inputs.suite }} path: /tmp/srcpkg-artifact - name: Extract prepared source tree @@ -74,12 +256,12 @@ jobs: - name: Build kernel package run: | - # Kernel source was cloned, patched, and prepared by the caller's - # prepare job. Pass --local-source and --skip-prepare so - # build-kernel.sh goes directly to the build step. - # debian/control, debian/changelog, and all config fragments are - # already baked into the extracted source tree; no re-derivation - # of LOCALVERSION, kver-extra, or debug config is needed here. + # Kernel source was cloned, patched, and prepared by the prepare job. + # Pass --local-source and --skip-prepare so build-kernel.sh goes + # directly to the build step. debian/control, debian/changelog, and + # all config fragments are already baked into the extracted source + # tree; no re-derivation of LOCALVERSION, kver-extra, or debug config + # is needed here. ./build-kernel.sh \ --build-mode docker \ --distro "$DISTRO" \ @@ -89,11 +271,11 @@ jobs: - name: Upload .deb packages to S3 # build-kernel.sh outputs to kernel-build// by default. - # Keep kernel variant and suite in the destination so concurrent matrix - # legs cannot overwrite or mix package outputs. Consumers must select - # the variant and suite they intend to install. + # Keep flavour and suite in the destination so concurrent matrix legs + # cannot overwrite or mix package outputs. Consumers must select the + # flavour and suite they intend to install. uses: qualcomm-linux/upload-private-artifact-action@aws-v4 with: s3_bucket: ${{ vars.ARTIFACT_S3_BUCKET }} - path: kernel-build/${{ inputs.distro }} - destination: ${{ github.repository_owner }}/pkg/temp/${{ github.event.repository.name }}/${{ inputs.flavour }}/${{ inputs.distro }}/${{ github.run_id }}-${{ github.run_attempt }}/ + path: kernel-build/${{ inputs.suite }} + destination: ${{ github.repository_owner }}/pkg/temp/${{ github.event.repository.name }}/${{ inputs.flavour }}/${{ inputs.suite }}/${{ github.run_id }}-${{ github.run_attempt }}/ diff --git a/.github/workflows/daily.yml b/.github/workflows/daily.yml index 06abc389..f78b97f3 100644 --- a/.github/workflows/daily.yml +++ b/.github/workflows/daily.yml @@ -9,9 +9,9 @@ name: daily # One entry is one generated package, so ci/scripts/resolve-matrix.py selects # the entries with type: Daily and passes them through as they stand. Every # field (name, suite, flavour, srcpkg, binpkg, kernel_config, dkms, -# debian_revision, git_clone, branch_or_tag, ref_strategy) becomes a -# build-kernel-deb.yml input. name labels the job; flavour is what reaches -# the built kernel's version and its published paths. +# debian_revision, git_clone, branch_or_tag, ref_strategy) becomes an input +# of the build workflow for the entry's family. name labels the job; flavour +# is what reaches the built kernel's version and its published paths. # # To add a daily build target: add an entry to ci/build-matrix.yaml with # type: Daily. No workflow changes needed. @@ -59,7 +59,8 @@ jobs: name: Configure build matrix runs-on: ubuntu-latest outputs: - matrix: ${{ steps.set-matrix.outputs.matrix }} + debian: ${{ steps.set-matrix.outputs.debian }} + ubuntu: ${{ steps.set-matrix.outputs.ubuntu }} steps: - uses: actions/checkout@v4 @@ -83,7 +84,7 @@ jobs: echo "Source: ci/build-matrix.yaml (full Daily matrix)" args=(--type Daily) ;; - "Selected variants") + "Selected builds") echo "Source: ci/build-matrix.yaml (builds: $INPUT_BUILDS)" args=(--type Daily --build "$INPUT_BUILDS") ;; @@ -96,27 +97,45 @@ jobs: exit 1 ;; esac - MATRIX=$(ci/scripts/resolve-matrix.py "${args[@]}") - echo "matrix=$MATRIX" >> "$GITHUB_OUTPUT" - echo "Matrix entries: $(echo "$MATRIX" | jq length)" + # One selection, split by build path. The two families are built by + # different workflows, so each gets the entries it builds and neither + # run starts a job for the other. A narrowed dispatch can leave one + # side empty, which --allow-empty reports as [] rather than as a + # failure; a name that matches nothing still fails, in both calls. + for family in debian ubuntu; do + MATRIX=$(ci/scripts/resolve-matrix.py "${args[@]}" \ + --family "$family" --allow-empty) + echo "$family=$MATRIX" >> "$GITHUB_OUTPUT" + echo "$family entries: $(echo "$MATRIX" | jq length)" + done # --------------------------------------------------------------------------- - # build: one parallel job per matrix entry. + # build-debian / build-ubuntu: one parallel job per matrix entry. # - # Each entry calls build-kernel-deb.yml as a reusable workflow. The full - # set of matrix fields (suite, srcpkg, binpkg, kernel_config, dkms, etc.) is - # threaded through as inputs. fail-fast is disabled so a failed trixie - # build does not cancel the forky build. + # Two jobs rather than one because the families are built by different + # workflows, on different runners, and published from different places. + # Splitting the matrix by family means each entry starts exactly the jobs + # that build it: a run shows no skipped job for the path it did not take. + # + # Both are named ${{ matrix.name }}, so the checks list reads the same as + # one job would -- the split is in what each entry calls, not in what it is + # called. fail-fast is disabled so a failed trixie build does not cancel the + # forky build. + # + # The if: guards only matter for a narrowed dispatch that selects nothing on + # one path; an empty matrix is an error rather than an empty job list. The + # scheduled full-matrix run has entries on both. # --------------------------------------------------------------------------- - build: + build-debian: name: ${{ matrix.name }} needs: configure-matrix + if: ${{ needs.configure-matrix.outputs.debian != '[]' }} strategy: fail-fast: false matrix: - include: ${{ fromJson(needs.configure-matrix.outputs.matrix) }} - uses: ./.github/workflows/build-kernel-deb.yml + include: ${{ fromJson(needs.configure-matrix.outputs.debian) }} + uses: ./.github/workflows/build-kernel-debian.yml with: suite: ${{ matrix.suite }} build: ${{ matrix.name }} @@ -136,4 +155,28 @@ jobs: secrets: DEBUSINE_USER: ${{ secrets.DEBUSINE_USER }} DEBUSINE_TOKEN: ${{ secrets.DEBUSINE_TOKEN }} - DEBUSINE_RELEASE_TOKEN: ${{ secrets.DEBUSINE_RELEASE_TOKEN }} + + build-ubuntu: + name: ${{ matrix.name }} + needs: configure-matrix + if: ${{ needs.configure-matrix.outputs.ubuntu != '[]' }} + strategy: + fail-fast: false + matrix: + include: ${{ fromJson(needs.configure-matrix.outputs.ubuntu) }} + uses: ./.github/workflows/build-kernel-ubuntu.yml + with: + suite: ${{ matrix.suite }} + build: ${{ matrix.name }} + flavour: ${{ matrix.flavour }} + git-clone: ${{ matrix.git_clone }} + kernel-branch: ${{ matrix.branch_or_tag }} + ref-strategy: ${{ matrix.ref_strategy }} + tag-pattern: ${{ matrix.tag_pattern || '' }} + srcpkg: ${{ matrix.srcpkg }} + binpkg: ${{ matrix.binpkg }} + kernel-config: ${{ matrix.kernel_config }} + dkms: ${{ matrix.dkms }} + debian-revision: ${{ matrix.debian_revision }} + localversion: ${{ matrix.localversion || '' }} + kver-extra: ${{ matrix.kver_extra || '' }} diff --git a/.github/workflows/pr-build.yml b/.github/workflows/pr-build.yml index df03c318..7b3f699a 100644 --- a/.github/workflows/pr-build.yml +++ b/.github/workflows/pr-build.yml @@ -30,7 +30,8 @@ jobs: name: Configure build matrix runs-on: ubuntu-latest outputs: - matrix: ${{ steps.set-matrix.outputs.matrix }} + debian: ${{ steps.set-matrix.outputs.debian }} + ubuntu: ${{ steps.set-matrix.outputs.ubuntu }} steps: - uses: actions/checkout@v4 with: @@ -41,24 +42,39 @@ jobs: id: set-matrix run: | set -euo pipefail - MATRIX=$(ci/scripts/resolve-matrix.py --type Daily) - echo "matrix=$MATRIX" >> "$GITHUB_OUTPUT" - echo "Matrix entries: $(echo "$MATRIX" | jq length)" + # Split by build path: the families are built by different workflows, + # so a PR gets one job per leg it actually builds and no skipped job + # for the path that leg does not take. + for family in debian ubuntu; do + MATRIX=$(ci/scripts/resolve-matrix.py --type Daily \ + --family "$family" --allow-empty) + echo "$family=$MATRIX" >> "$GITHUB_OUTPUT" + echo "$family entries: $(echo "$MATRIX" | jq length)" + done # --------------------------------------------------------------------------- - # build: one parallel job per matrix entry, mirroring daily.yml. + # build-debian / build-ubuntu: one parallel job per matrix entry, mirroring + # daily.yml. + # + # A PR gets one check per build leg and nothing else: neither workflow has a + # branch to skip, and a PR builds the whole Daily matrix, so both families + # have entries and neither guard fires. # # fail-fast is disabled so a broken trixie leg still leaves forky and # resolute results on the PR. + # + # Neither takes DEBUSINE_RELEASE_TOKEN: a PR build never promotes, so it + # calls the S3-publishing workflow and not release-kernel-debian.yml. # --------------------------------------------------------------------------- - build: + build-debian: name: ${{ matrix.name }} needs: configure-matrix + if: ${{ needs.configure-matrix.outputs.debian != '[]' }} strategy: fail-fast: false matrix: - include: ${{ fromJson(needs.configure-matrix.outputs.matrix) }} - uses: ./.github/workflows/build-kernel-deb.yml + include: ${{ fromJson(needs.configure-matrix.outputs.debian) }} + uses: ./.github/workflows/build-kernel-debian.yml with: suite: ${{ matrix.suite }} build: ${{ matrix.name }} @@ -77,7 +93,30 @@ jobs: # in the shared Debusine workspace. kver-extra: -pr${{ github.event.number }} secrets: - # No DEBUSINE_RELEASE_TOKEN: PR builds never promote to a release - # workspace, so they leave target-workspace empty. DEBUSINE_USER: ${{ secrets.DEBUSINE_USER }} DEBUSINE_TOKEN: ${{ secrets.DEBUSINE_TOKEN }} + + build-ubuntu: + name: ${{ matrix.name }} + needs: configure-matrix + if: ${{ needs.configure-matrix.outputs.ubuntu != '[]' }} + strategy: + fail-fast: false + matrix: + include: ${{ fromJson(needs.configure-matrix.outputs.ubuntu) }} + uses: ./.github/workflows/build-kernel-ubuntu.yml + with: + suite: ${{ matrix.suite }} + build: ${{ matrix.name }} + flavour: ${{ matrix.flavour }} + git-clone: ${{ matrix.git_clone }} + kernel-branch: ${{ matrix.branch_or_tag }} + ref-strategy: ${{ matrix.ref_strategy }} + tag-pattern: ${{ matrix.tag_pattern || '' }} + srcpkg: ${{ matrix.srcpkg }} + binpkg: ${{ matrix.binpkg }} + kernel-config: ${{ matrix.kernel_config }} + dkms: ${{ matrix.dkms }} + debian-revision: ${{ matrix.debian_revision }} + localversion: ${{ matrix.localversion || '' }} + kver-extra: -pr${{ github.event.number }} diff --git a/.github/workflows/release-kernel-debian.yml b/.github/workflows/release-kernel-debian.yml new file mode 100644 index 00000000..759d054b --- /dev/null +++ b/.github/workflows/release-kernel-debian.yml @@ -0,0 +1,210 @@ +name: release-kernel-debian + +# Build a kernel package for a Debian-family suite and promote it to a release +# workspace in Debusine. +# +# Called per matrix entry by release.yml. Identical to build-kernel-debian.yml +# up to the Debusine build -- both share the prepare-kernel-source and +# debusine-build actions -- and differs only in the tail: this one promotes the +# built package to target-workspace rather than publishing it to the daily S3 +# path. Which tail a run wants is decided by the workflow that calls it, so +# neither file carries the other's tail as a job to be skipped. +# +# Only the Debian family has a release path: promotion runs through Debusine, +# and resolve-matrix.py rejects a Release entry for any other suite. +# +# Jobs: +# prepare (self-hosted): clone, patch, inject debian/, generate the source. +# build (debusine-pkg-builder container): submit to Debusine. +# release (debusine-pkg-builder container): promote to the target workspace. + +on: + workflow_call: + inputs: + build: + description: 'Name of this build, as listed in ci/build-matrix.yaml' + type: string + default: 'qcom-next-trixie' + flavour: + description: 'Kernel flavour: the LOCALVERSION suffix and so the kernel release identity' + type: string + default: 'qcom-next' + suite: + description: 'Target Debian suite (trixie, forky, sid)' + type: string + default: 'trixie' + kernel-branch: + description: 'Immutable ref to build, from the entry branch_or_tag' + type: string + default: 'qcom-next' + git-clone: + description: 'Kernel repository URL' + type: string + default: 'https://github.com/qualcomm-linux/kernel' + ref-strategy: + description: 'Kernel ref strategy: a Release entry always pins one' + type: string + default: 'pinned_ref' + tag-pattern: + description: 'Tag glob, unused by pinned_ref and carried for symmetry with the Daily inputs' + type: string + default: '' + srcpkg: + description: 'Source package name' + type: string + default: 'linux-qcom-next' + binpkg: + description: 'Binary metapackage name' + type: string + default: 'linux-image-qcom-next' + kernel-config: + description: 'Extra config fragments applied on top of all of debian/config-available/' + type: string + default: '' + dkms: + description: 'Out-of-tree DKMS modules to bundle, comma-separated and without the -dkms suffix' + type: string + default: '' + debian-revision: + description: 'Debian revision component of the package version' + type: string + default: '0qli' + localversion: + description: 'Override LOCALVERSION suffix (auto-derived from the resolved ref if empty)' + type: string + default: '' + kver-extra: + description: 'Extra suffix appended to the package version' + type: string + default: '' + debusine-parent-workspace: + description: 'Parent Debusine workspace for CI child workspace creation' + type: string + default: '' + target-workspace: + description: 'Debusine workspace to promote the built package into' + type: string + required: true + secrets: + DEBUSINE_USER: + description: 'Debusine account used to submit release builds' + required: false + DEBUSINE_TOKEN: + description: 'Debusine token for the CI child workspace' + required: false + DEBUSINE_RELEASE_TOKEN: + description: 'Debusine token for promotion to the release target workspace' + required: false + +permissions: + contents: read + packages: read + +jobs: + # --------------------------------------------------------------------------- + # prepare: clone kernel source, inject packaging, run prepare-source.sh. + # --------------------------------------------------------------------------- + prepare: + name: Prepare kernel source + runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] + steps: + - name: Checkout pkg-linux-qcom + uses: actions/checkout@v4 + + - name: Prepare kernel source + uses: ./.github/actions/prepare-kernel-source + with: + build: ${{ inputs.build }} + suite: ${{ inputs.suite }} + expect-family: debian + flavour: ${{ inputs.flavour }} + kernel-url: ${{ inputs.git-clone }} + ref-strategy: ${{ inputs.ref-strategy }} + kernel-branch: ${{ inputs.kernel-branch }} + tag-pattern: ${{ inputs.tag-pattern }} + srcpkg: ${{ inputs.srcpkg }} + binpkg: ${{ inputs.binpkg }} + kernel-config: ${{ inputs.kernel-config }} + dkms: ${{ inputs.dkms }} + debian-revision: ${{ inputs.debian-revision }} + localversion: ${{ inputs.localversion }} + kver-extra: ${{ inputs.kver-extra }} + + # --------------------------------------------------------------------------- + # build: generate the Debian source package and submit it to Debusine. + # --------------------------------------------------------------------------- + build: + name: Build (Debusine) + needs: prepare + runs-on: ubuntu-latest + environment: Production + container: + image: ghcr.io/qualcomm-linux/debusine-pkg-builder:trixie + options: --user 0:0 + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + outputs: + workspace: ${{ steps.debusine.outputs.workspace }} + srcpkg_version: ${{ steps.debusine.outputs.srcpkg_version }} + steps: + - name: Checkout pkg-linux-qcom + uses: actions/checkout@v4 + + - name: Build in Debusine + id: debusine + uses: ./.github/actions/debusine-build + with: + suite: ${{ inputs.suite }} + flavour: ${{ inputs.flavour }} + debusine-host: ${{ vars.DEBUSINE_HOST }} + debusine-scope: ${{ vars.DEBUSINE_SCOPE }} + debusine-user: ${{ secrets.DEBUSINE_USER }} + debusine-token: ${{ secrets.DEBUSINE_TOKEN }} + debusine-parent-workspace: ${{ inputs.debusine-parent-workspace || vars.DEBUSINE_PARENT_WORKSPACE || 'qli-ci' }} + # A release build reads qli alone, so a released kernel is never + # built against something that has not been released alongside it. + extra-build-dep-workspaces: qli + + # --------------------------------------------------------------------------- + # release: promote the built package from the CI workspace to the target. + # --------------------------------------------------------------------------- + release: + name: Release to Debusine + needs: build + runs-on: ubuntu-latest + environment: Production + container: + image: ghcr.io/qualcomm-linux/debusine-pkg-builder:trixie + options: --user 0:0 + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + defaults: + run: + shell: bash + steps: + - name: Checkout debusine-action helpers + uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5 + with: + persist-credentials: false + repository: qualcomm-linux/debusine-action + ref: main + path: debusine-action + fetch-depth: 1 + sparse-checkout: | + lib + + - name: Promote packages to target workspace + env: + DEBUSINE_HOST: ${{ vars.DEBUSINE_HOST }} + DEBUSINE_SCOPE: ${{ vars.DEBUSINE_SCOPE }} + DEBUSINE_TOKEN: ${{ secrets.DEBUSINE_RELEASE_TOKEN }} + DEBUSINE_CI_WORKSPACE: ${{ needs.build.outputs.workspace }} + DEBUSINE_TARGET_WORKSPACE: ${{ inputs.target-workspace }} + SRCPKG_NAME: ${{ inputs.srcpkg }} + SRCPKG_VERSION: ${{ needs.build.outputs.srcpkg_version }} + SUITE: ${{ inputs.suite }} + run: | + set -ex + debusine-action/lib/release diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index e2e55206..bdbbb1e1 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -71,7 +71,7 @@ jobs: echo "Source: ci/build-matrix.yaml (flavour: $INPUT_FLAVOUR, all suites)" args=(--type Release --flavour "$INPUT_FLAVOUR") ;; - "Selected variants") + "Selected builds") echo "Source: ci/build-matrix.yaml (builds: $INPUT_BUILDS)" args=(--type Release --build "$INPUT_BUILDS") ;; @@ -89,8 +89,14 @@ jobs: # build: one parallel job per Release matrix entry. # # Uses the ref strategy and branch_or_tag selected by the matrix. - # Passes target_workspace so build-kernel-debusine.yml triggers the Debusine - # release path (lib/release) instead of the S3 publish path. + # release-kernel-debian.yml builds exactly as the daily workflow does and + # then promotes to target_workspace, rather than publishing to the daily S3 + # path. Choosing the workflow here is what makes that the whole of the run: + # no job in it exists to publish the other way and be skipped. + # + # There is no Ubuntu counterpart. Promotion runs through Debusine, which + # builds only the Debian family, so resolve-matrix.py rejects a Release entry + # for any other suite and this matrix is Debian by construction. # --------------------------------------------------------------------------- build: name: ${{ matrix.name }} @@ -99,7 +105,7 @@ jobs: fail-fast: false matrix: include: ${{ fromJson(needs.configure-matrix.outputs.matrix) }} - uses: ./.github/workflows/build-kernel-deb.yml + uses: ./.github/workflows/release-kernel-debian.yml with: suite: ${{ matrix.suite }} build: ${{ matrix.name }} diff --git a/README.md b/README.md index eca38a6d..3fefeeb2 100644 --- a/README.md +++ b/README.md @@ -129,9 +129,24 @@ variant is a matrix change, not a workflow redesign. | --- | --- | --- | | `daily.yml` | Resolves and runs the Daily matrix. | Scheduled daily at `23:00 UTC`, or manual dispatch. | | `release.yml` | Resolves and runs the Release matrix. | Manual dispatch only. | -| `build-kernel-deb.yml` | Reusable orchestrator for one kernel variant and suite. | Manual dispatch or called by Daily and Release. | -| `build-kernel-debusine.yml` | Builds Debian suites in Debusine and either publishes Daily artifacts or promotes Releases. | Called by `build-kernel-deb.yml`. | -| `build-kernel-ubuntu.yml` | Builds Ubuntu-family suites with the Docker path. | Called by `build-kernel-deb.yml`. | +| `build-kernel-debian.yml` | Builds one Debian-suite entry in Debusine and publishes it to S3. | Manual dispatch or called by Daily and PR build. | +| `build-kernel-ubuntu.yml` | Builds one Ubuntu-suite entry on the Docker path and publishes it to S3. | Manual dispatch or called by Daily and PR build. | +| `release-kernel-debian.yml` | Builds one Debian-suite entry in Debusine and promotes it to the release workspace. | Called by Release. | + +The three build workflows share their steps through two composite actions +rather than through a common orchestrator workflow: + +| Action | Used by | +| --- | --- | +| `.github/actions/prepare-kernel-source` | All three, as the `prepare` job. | +| `.github/actions/debusine-build` | The two Debian workflows, as the `build` job. | + +Which of them a build leg calls is decided by the caller, from the entry's +suite: `resolve-matrix.py --family debian|ubuntu` splits the selection, and +each family's entries call only the workflow that builds them. Nothing inside a +build workflow is conditional on the suite or on whether the run releases, so a +run starts exactly the jobs it needs and shows no skipped job for a path it did +not take. ### Daily @@ -171,9 +186,15 @@ Release is the controlled promotion path. the release credential and enforces the required approval gate before promotion to `qli`. -Direct `build-kernel-deb.yml` dispatches are build-only. Release promotion is +Direct `build-kernel-debian.yml` and `build-kernel-ubuntu.yml` dispatches are +build-only: neither has a promotion path to offer. Release promotion is initiated exclusively by `release.yml`, which owns the target workspace and -production release controls. +production release controls, and is the only caller of +`release-kernel-debian.yml`. + +Only the Debian family has a release path at all, because promotion runs +through Debusine. `resolve-matrix.py` rejects a `Release` entry for any other +suite rather than letting it build and then silently not promote. ## Matrix Model @@ -236,10 +257,11 @@ face value: - Where a suite has both, its `Daily` revision is its `Release` revision plus a trailing `~`. -`build-kernel-deb.yml`'s direct-dispatch path has no matrix context of its own, -so when its `debian-revision` input is empty it looks up the `Daily` entry for -the variant and suite it was given (`resolve-matrix.py --field -debian_revision`) and builds at the revision the daily build would have used. +A direct dispatch of a build workflow has no matrix context of its own, so when +its `debian-revision` input is empty the `prepare-kernel-source` action looks up +the `Daily` entry for the build name and suite it was given (`resolve-matrix.py +--field debian_revision`) and builds at the revision the daily build would have +used. Each entry has a distinct prepared-source artifact, Debusine child workspace, and S3 path keyed by `flavour + suite`. This prevents two flavours that both @@ -277,16 +299,22 @@ This document covers the CI generator. For the packaging internals: `debian/rule targets, the config fragment merge pipeline, DKMS module bundling and the produced package layout see [debian/README.md](debian/README.md). +Both branches below are taken in the caller, when the matrix is resolved: the +family from the entry's suite, the tail from which workflow is running. By the +time a build workflow starts, there is nothing left to decide. + ```mermaid flowchart LR - IN["Matrix variant + suite input"] --> R{"Resolve suite family"} + IN["Matrix entries"] --> R{"resolve-matrix.py\n--family"} - R -->|"trixie · forky"| DEB["Debian path\nbuild-kernel-debusine.yml\nGenerate source package\nSubmit with lib/build\nDebusine builds binaries"] - R -->|"resolute"| UBU["Ubuntu path\nbuild-kernel-ubuntu.yml\nbuild-kernel.sh in Docker\nBuild binary packages"] + R -->|"debian: trixie · forky"| DT{"Which caller"} + R -->|"ubuntu: resolute"| UBU["build-kernel-ubuntu.yml\nbuild-kernel.sh in Docker\nBuild binary packages"] - DEB --> DOUT{"Build type"} - DOUT -->|Daily| S3["Download .deb files\nPublish to S3"] - DOUT -->|Release| QLI["Promote source and binaries\nto qli"] + DT -->|"daily.yml · pr-build.yml"| DEB["build-kernel-debian.yml\nGenerate source package\nSubmit with lib/build\nDebusine builds binaries"] + DT -->|"release.yml"| REL["release-kernel-debian.yml\nSame build, release tail"] + + DEB --> S3["Download .deb files\nPublish to S3"] + REL --> QLI["Promote source and binaries\nto qli"] UBU --> US3["Publish .deb files to S3"] ``` @@ -298,23 +326,26 @@ flowchart LR flowchart TD subgraph triggers[Triggers] A1["daily.yml\nScheduled full matrix"] - A2["daily.yml\nManual full or filtered variant + suite"] - A3["release.yml\nManual full or filtered variant + suite"] - A4["build-kernel-deb.yml\nManual one-off build"] + A2["daily.yml\nManual full or filtered builds"] + A5["pr-build.yml\nFull Daily matrix on every PR"] + A3["release.yml\nManual full or filtered builds"] + A4["build-kernel-debian.yml\nbuild-kernel-ubuntu.yml\nManual one-off build"] end subgraph matrix[Matrix entry points] - B1["Daily configure-matrix\nFlatten Daily rows"] - B2["Daily variant + suite legs\nqcom-next / trixie · forky · resolute\nqcom-next-debug / trixie · forky"] - B3["Release configure-matrix\nFlatten Release rows"] - B4["Release variant + suite legs\nqcom-next / trixie · forky\nqcom-next-debug / trixie · forky"] + B1["configure-matrix\nDaily entries, split by family"] + B2["build-debian legs\nqcom-next · qcom-next-debug · qcom-arduino\nmainline · next / trixie · forky"] + B5["build-ubuntu legs\nqcom-next / resolute"] + B3["configure-matrix\nRelease entries, Debian by construction"] + B4["build legs\nqcom-next · qcom-next-debug / trixie · forky"] end - subgraph orchestrator[build-kernel-deb.yml] - C1["resolve\nClassify suite family"] - C2["prepare\nClone selected kernel ref\nRun prepare-source.sh\nUpload kernel-srcpkg-variant-suite"] - C3["debusine-build\nDebian suites only"] - C4["ubuntu-build\nUbuntu suites only"] + subgraph build[One build workflow per leg] + C2["prepare\nprepare-kernel-source action\nClone ref, run prepare-source.sh\nUpload kernel-srcpkg-flavour-suite"] + C3["build\ndebusine-build action"] + C4["build\nbuild-kernel.sh in Docker"] + C5["publish\nDownload .deb files, upload to S3"] + C6["release\nPromote to target workspace"] end subgraph outputs[Outputs] @@ -324,16 +355,25 @@ flowchart TD A1 --> B1 A2 --> B1 + A5 --> B1 A3 --> B3 - B1 --> B2 --> C1 - B3 --> B4 --> C1 - A4 --> C1 - C1 --> C2 + B1 --> B2 & B5 + B3 --> B4 + B2 --> C2 + B5 --> C2 + B4 --> C2 + A4 --> C2 C2 --> C3 & C4 - C3 --> D1 & D2 + C3 --> C5 & C6 C4 --> D1 + C5 --> D1 + C6 --> D2 ``` +Every leg runs every job drawn under it. `build-debian` and `build-ubuntu` legs +reach different build workflows, and `publish` and `release` belong to +different ones, so no leg starts a job it will skip. + ### Prepare stage ```mermaid @@ -452,15 +492,18 @@ out-of-tree module builds are required. ## Manual Builds -Use **Actions** → **build-kernel-deb** for a one-off build. It is an explicit -override workflow, not a matrix-derived delivery flow: use `daily.yml` and -`release.yml` for normal Daily and Release operations. +Use **Actions** → **build-kernel-debian** or **build-kernel-ubuntu** for a +one-off build, picking the one that builds the suite you want. These are +explicit override workflows, not matrix-derived delivery flows: use `daily.yml` +and `release.yml` for normal Daily and Release operations. Neither promotes. `build`, `suite`, and `ref-strategy` are the required build selection. All remaining package, configuration, and PR inputs are advanced overrides for -validation or debugging. Variant and suite are free-text matrix values rather -than static dropdowns, so adding a matrix entry never requires editing the -workflow UI. +validation or debugging. Build name and suite are free-text matrix values +rather than static dropdowns, so adding a matrix entry never requires editing +the workflow UI. A suite belonging to the other family is rejected by the +`prepare` job before anything is cloned or built, so the only cost of picking +the wrong workflow is a fast failure. The available inputs are: diff --git a/ci/scripts/resolve-matrix.py b/ci/scripts/resolve-matrix.py index 3f5082ec..8151cfdf 100755 --- a/ci/scripts/resolve-matrix.py +++ b/ci/scripts/resolve-matrix.py @@ -18,6 +18,7 @@ ci/scripts/resolve-matrix.py --type Daily --build qcom-next-trixie ci/scripts/resolve-matrix.py --type Daily --build qcom-next-trixie,qcom-next-forky ci/scripts/resolve-matrix.py --type Release --flavour qcom-next + ci/scripts/resolve-matrix.py --type Daily --family ubuntu --allow-empty ci/scripts/resolve-matrix.py --type Daily --build qcom-next-trixie \\ --field debian_revision @@ -31,6 +32,14 @@ flavour spans its suites, so this asks for one kernel everywhere it is built. --suite SUITES Select only these suites, comma-separated. + --family FAMILY Select only the builds taking one build path, + debian or ubuntu. Derived from suite, so it + selects by how a build is built rather than by + naming every suite that is built that way. + --allow-empty Print [] rather than failing when the filters + select nothing. For a caller asking each family + for the same selection, where one of them + having nothing to build is an ordinary answer. --field NAME Print just this field of the single selected entry, unquoted. Errors unless exactly one entry matches. @@ -39,12 +48,14 @@ Filters combine: an entry must match every filter given. Every name in a filter must match at least one entry of the selected type, so a typo or a - stale name fails instead of quietly narrowing the build set. + stale name fails instead of quietly narrowing the build set. That check is + per filter, so --allow-empty still rejects a name that matches nothing. Output: Without --field, a compact JSON array of the selected entries, ready for a - GitHub Actions matrix `include`. kernel_config and dkms are joined into the - comma-separated strings that build-kernel-deb.yml's kernel-config and dkms + GitHub Actions matrix `include`. Each entry carries a derived family field + naming its build path. kernel_config and dkms are joined into the + comma-separated strings that the build workflows' kernel-config and dkms inputs, and prepare-source.sh's --kernel-config and --dkms, expect; every other field is passed through as written. @@ -76,6 +87,20 @@ DELIVERY_TYPES = ("Daily", "Release") REF_STRATEGIES = ("latest_tag", "branch_tip", "pinned_ref") +# Which build path a suite takes. Debian-family suites are built by Debusine; +# everything else is built in a suite-matched docker container on the +# self-hosted runner. The two paths are different workflows with different +# runners, containers and publish steps, so a caller selects entries by family +# and calls the one workflow that builds them -- rather than every build leg +# starting both and skipping one. +DEBIAN_SUITES = ("trixie", "forky", "sid", "unstable", "bookworm") +FAMILIES = ("debian", "ubuntu") + + +def family_for(suite): + """Return the build family a suite belongs to.""" + return "debian" if suite in DEBIAN_SUITES else "ubuntu" + # A delivery type constrains how its kernel ref is chosen: a Daily build tracks # something moving, a Release build is pinned to an immutable ref. REF_STRATEGIES_FOR_TYPE = { @@ -272,6 +297,17 @@ def check_entry(entry, report): if delivery_type == "Release": if not entry.get("target_workspace"): report("missing or invalid target_workspace") + # Promotion runs through Debusine, and only the Debian family is built + # there. An Ubuntu Release entry would build the package and then have + # no workspace to promote it from, publishing to the daily S3 path and + # reporting success without ever releasing anything. + suite = entry.get("suite") + if isinstance(suite, str) and family_for(suite) != "debian": + report( + f"Release entries must target a Debian suite, not {suite}; " + "promotion runs through Debusine, which builds " + + ", ".join(DEBIAN_SUITES) + ) elif "target_workspace" in entry: report("target_workspace is only valid for Release") @@ -489,6 +525,12 @@ def load_matrix(path): + "\n".join(f" - {error}" for error in errors) ) + # Derived, never written: family follows from suite, so the matrix cannot + # state one that disagrees with the suite it is built for. Attached after + # validation, which rejects family as an unknown field on an entry. + for entry in builds: + entry["family"] = family_for(entry["suite"]) + return builds @@ -517,10 +559,16 @@ def select(builds, delivery_type, filters): def unmatched_filters(builds, delivery_type, filters): - """Names asked for that no entry of this delivery type offers.""" + """Names asked for that no entry of this delivery type offers. + + family is exempt: it routes a selection to a build path rather than naming + something in the matrix, argparse already restricts it to a real family, + and a type having no builds on one path is an ordinary state of the matrix + rather than a mistake in the request. + """ missing = [] for field, wanted in filters.items(): - if wanted is None: + if wanted is None or field == "family": continue available = { entry[field] for entry in builds if entry["type"] == delivery_type @@ -573,6 +621,17 @@ def main(): parser.add_argument( "--suite", default="", help="select only these suites, comma-separated" ) + parser.add_argument( + "--family", + default="", + choices=("",) + FAMILIES, + help="select only the builds taking this build path", + ) + parser.add_argument( + "--allow-empty", + action="store_true", + help="print [] instead of failing when the filters select nothing", + ) parser.add_argument( "--field", default="", @@ -588,6 +647,7 @@ def main(): "name": parse_filter(args.build), "flavour": parse_filter(args.flavour), "suite": parse_filter(args.suite), + "family": parse_filter(args.family), } what = describe_selection(args.type, filters) @@ -603,6 +663,13 @@ def main(): selected = select(builds, args.type, filters) if not selected: + # A caller splitting one dispatch across both build paths asks each + # family for the same selection, and one of them legitimately has + # nothing to build. Every name in the request still had to match + # something above, so this is an empty intersection, not a typo. + if args.allow_empty and not args.field: + print("[]") + return sys.exit(f"ERROR: No matrix entries found for {what}") if not args.field: From a845d399bde3d84f23afe6065953e24ad0ca8df2 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 11/59] ci: pass prepare's values as step outputs and drop checkout credentials zizmor raised two findings against the new action and workflows. prepare-kernel-source wrote its intermediate values to GITHUB_ENV, which puts them in the environment of every later step in the job rather than only the steps that want them, and lets a value containing a newline define environment variables of its own -- LD_PRELOAD or NODE_OPTIONS being the interesting ones. Not all of these values are ours to trust: the kernel ref is whatever the remote's tag list offers, and on pr-build.yml the inputs come from the build matrix as the pull request wrote it, since a PR is built against its own ci/build-matrix.yaml. So the steps now hand their values on as step outputs, and each consumer names in its own env: block what it reads. That is the same data flowing the same way, but scoped to the steps that asked for it. The resolved kernel ref is also rejected outright if it contains a newline, because that one is chosen furthest from us and step outputs are injectable in the same manner if less usefully. The checkouts leave the GitHub token in .git/config, which the prepare job then packs a sibling directory into an artifact next to. None of these checkouts is fetched from or pushed to afterwards: the kernel is cloned separately with its own remote, and the PR-merge steps fetch inside that clone rather than this one. So they all set persist-credentials: false, as release-kernel-debian.yml's debusine-action checkout and pr-build.yml and release.yml already did. daily.yml's configure-matrix checkout gets the same treatment. zizmor did not flag it, having only reviewed the diff, but it is the same checkout doing the same job as the two that were already set that way. No build changes what it produces, and the job graph is as it was: 32 jobs across 11 legs for a daily or PR run, 12 across 4 for a release, none skipped. Signed-off-by: Christopher Obbard --- .github/actions/debusine-build/action.yml | 1 + .../actions/prepare-kernel-source/action.yml | 116 +++++++++++++++--- .github/workflows/build-kernel-debian.yml | 5 + .github/workflows/build-kernel-ubuntu.yml | 5 + .github/workflows/daily.yml | 2 + .github/workflows/release-kernel-debian.yml | 4 + 6 files changed, 117 insertions(+), 16 deletions(-) diff --git a/.github/actions/debusine-build/action.yml b/.github/actions/debusine-build/action.yml index 0eff3ade..6355aea6 100644 --- a/.github/actions/debusine-build/action.yml +++ b/.github/actions/debusine-build/action.yml @@ -66,6 +66,7 @@ runs: - name: Checkout debusine-action helpers uses: actions/checkout@v5 with: + persist-credentials: false repository: qualcomm-linux/debusine-action ref: ${{ inputs.debusine-action-ref }} path: debusine-action diff --git a/.github/actions/prepare-kernel-source/action.yml b/.github/actions/prepare-kernel-source/action.yml index ae171847..bfe94518 100644 --- a/.github/actions/prepare-kernel-source/action.yml +++ b/.github/actions/prepare-kernel-source/action.yml @@ -11,8 +11,14 @@ description: > # The caller must have checked out pkg-linux-qcom into the workspace first: # debian/ and ci/ come from that checkout, and so does this action. # -# Runs on the self-hosted arm64 runner. Leaves KERNEL_DIR, KERNEL_REF, -# KERNEL_SHA, LOCALVERSION and DEBIAN_REVISION in the job environment. +# Runs on the self-hosted arm64 runner, and leaves nothing behind in the job +# environment: values pass between these steps as step outputs, each consumer +# naming what it reads. Writing them to GITHUB_ENV instead would put them in +# the environment of every later step in the job, including steps outside this +# action, and a newline in one would let the value that carried it define +# environment variables of its own. Several of these values are not ours: the +# kernel ref comes from a remote's tag list, and on pr-build.yml the inputs +# come from the build matrix as the pull request wrote it. inputs: build: @@ -77,6 +83,7 @@ runs: using: composite steps: - name: Resolve build environment + id: env shell: bash env: SUITE_INPUT: ${{ inputs.suite }} @@ -103,24 +110,28 @@ runs: exit 1 fi - echo "DISTRO=$SUITE_INPUT" >> "$GITHUB_ENV" - echo "DOCKER_DISTRO=$DOCKER_DISTRO" >> "$GITHUB_ENV" + echo "distro=$SUITE_INPUT" >> "$GITHUB_OUTPUT" + echo "docker_distro=$DOCKER_DISTRO" >> "$GITHUB_OUTPUT" echo "Suite $SUITE_INPUT is $FAMILY-family and builds in the $DOCKER_DISTRO image" - name: Checkout docker-pkg-build uses: actions/checkout@v4 with: + persist-credentials: false repository: qualcomm-linux/docker-pkg-build ref: main path: docker-pkg-build - name: Build docker image shell: bash + env: + DOCKER_DISTRO: ${{ steps.env.outputs.docker_distro }} run: | ./docker-pkg-build/docker_deb_build.py --rebuild -d "$DOCKER_DISTRO" docker image ls - name: Resolve kernel ref + id: ref shell: bash env: KERNEL_URL_INPUT: ${{ inputs.kernel-url }} @@ -134,8 +145,6 @@ runs: echo "::error::build name must use lowercase letters, digits, and internal hyphens" exit 1 } - echo "KERNEL_URL=$KERNEL_URL_INPUT" >> "$GITHUB_ENV" - case "$REF_STRATEGY_INPUT" in latest_tag) KERNEL_REF=$(ci/scripts/resolve-kernel-ref.sh \ @@ -152,41 +161,91 @@ runs: ;; esac - echo "KERNEL_REF=$KERNEL_REF" >> "$GITHUB_ENV" + # A ref carrying a newline would otherwise define outputs of its own. + # The remote chooses this value, so check it before writing it out. + [[ "$KERNEL_REF" == *$'\n'* ]] && { + echo "::error::resolved kernel ref contains a newline" + exit 1 + } + + echo "kernel_ref=$KERNEL_REF" >> "$GITHUB_OUTPUT" echo "Build: $BUILD_INPUT" echo "Ref strategy: $REF_STRATEGY_INPUT" echo "Resolved kernel ref: $KERNEL_REF" - name: Clone kernel source + id: clone shell: bash + env: + KERNEL_URL: ${{ inputs.kernel-url }} + KERNEL_REF: ${{ steps.ref.outputs.kernel_ref }} run: | set -euo pipefail KERNEL_DIR="$GITHUB_WORKSPACE/kernel-source" - echo "KERNEL_DIR=$KERNEL_DIR" >> "$GITHUB_ENV" + echo "kernel_dir=$KERNEL_DIR" >> "$GITHUB_OUTPUT" echo "Cloning $KERNEL_URL @ $KERNEL_REF (shallow)..." git clone --depth 1 --single-branch --branch "$KERNEL_REF" --no-tags \ "$KERNEL_URL" "$KERNEL_DIR" KERNEL_SHA=$(git -C "$KERNEL_DIR" rev-parse HEAD) - echo "KERNEL_SHA=$KERNEL_SHA" >> "$GITHUB_ENV" + echo "kernel_sha=$KERNEL_SHA" >> "$GITHUB_OUTPUT" echo "Kernel HEAD SHA: $KERNEL_SHA" - - name: Derive LOCALVERSION + # Committer date of HEAD, normalised to UTC. Committer rather than + # author date: an author date can be months old on a backported patch, + # which would date the build by when someone first wrote the code + # rather than by when this snapshot came to exist. + KERNEL_DATE=$(TZ=UTC git -C "$KERNEL_DIR" log -1 --format=%cd --date=format-local:%Y%m%d) + [[ "$KERNEL_DATE" =~ ^[0-9]{8}$ ]] || { + echo "::error::could not read a YYYYMMDD commit date from $KERNEL_REF (got '$KERNEL_DATE')" + exit 1 + } + echo "kernel_date=$KERNEL_DATE" >> "$GITHUB_OUTPUT" + echo "Kernel HEAD commit date (UTC): $KERNEL_DATE" + + - name: Derive version fields + id: localversion shell: bash env: LOCALVERSION_INPUT: ${{ inputs.localversion }} FLAVOUR_INPUT: ${{ inputs.flavour }} + KERNEL_REF: ${{ steps.ref.outputs.kernel_ref }} + KERNEL_SHA: ${{ steps.clone.outputs.kernel_sha }} + KERNEL_DATE: ${{ steps.clone.outputs.kernel_date }} run: | set -euo pipefail if [[ -n "$LOCALVERSION_INPUT" ]]; then + # An override supplies no snapshot, and the ref it was chosen to + # replace cannot be trusted to describe it. The Debian version then + # carries no +git; prepare-source.sh warns about the same. echo "Using explicit LOCALVERSION override: $LOCALVERSION_INPUT" - echo "LOCALVERSION=$LOCALVERSION_INPUT" >> "$GITHUB_ENV" + echo "localversion=$LOCALVERSION_INPUT" >> "$GITHUB_OUTPUT" + echo "snapshot=" >> "$GITHUB_OUTPUT" + echo "gitsha=" >> "$GITHUB_OUTPUT" else - LOCALVERSION=$(ci/scripts/derive-localversion.sh \ + # Emits LOCALVERSION=, SNAPSHOT= and GITSHA= lines. They are read + # here rather than recovered from LOCALVERSION later: that string + # also carries a flavour name, and a hex SHA can end in eight digits + # of its own. + # + # --date is only consulted for branch-tip builds, where the ref + # carries no snapshot. It comes from the commit rather than the + # clock, so rebuilding a commit reproduces its version instead of + # inventing a higher one. + FIELDS=$(ci/scripts/derive-localversion.sh \ --flavour "$FLAVOUR_INPUT" \ --ref "$KERNEL_REF" \ - --sha "$(echo "$KERNEL_SHA" | cut -c1-12)") - echo "LOCALVERSION=$LOCALVERSION" >> "$GITHUB_ENV" - echo "Derived LOCALVERSION: $LOCALVERSION" + --sha "$KERNEL_SHA" \ + --date "$KERNEL_DATE") + echo "Derived version fields:" + echo "$FIELDS" + echo "$FIELDS" | while IFS='=' read -r key value; do + case "$key" in + LOCALVERSION) echo "localversion=$value" ;; + SNAPSHOT) echo "snapshot=$value" ;; + GITSHA) echo "gitsha=$value" ;; + *) echo "::error::unexpected field '$key' from derive-localversion.sh"; exit 1 ;; + esac + done >> "$GITHUB_OUTPUT" fi - name: Resolve Debian revision @@ -196,10 +255,12 @@ runs: # exactly the revision the daily build would have used. The lookup is # always the Daily entry: direct dispatch is build-only and never # promotes. + id: revision shell: bash env: DEBIAN_REVISION_INPUT: ${{ inputs.debian-revision }} BUILD_INPUT: ${{ inputs.build }} + DISTRO: ${{ steps.env.outputs.distro }} run: | set -euo pipefail if [[ -n "$DEBIAN_REVISION_INPUT" ]]; then @@ -217,13 +278,14 @@ runs: } echo "Matrix Debian revision for $BUILD_INPUT/$DISTRO: $DEBIAN_REVISION" fi - echo "DEBIAN_REVISION=$DEBIAN_REVISION" >> "$GITHUB_ENV" + echo "debian_revision=$DEBIAN_REVISION" >> "$GITHUB_OUTPUT" - name: Merge qcom-next PR patches if: ${{ inputs.qcom-next-pr != '' }} shell: bash env: QCOM_NEXT_PR: ${{ inputs.qcom-next-pr }} + KERNEL_DIR: ${{ steps.clone.outputs.kernel_dir }} run: | cd "$KERNEL_DIR" echo "Merging qcom-next PR(s): $QCOM_NEXT_PR" @@ -250,6 +312,7 @@ runs: shell: bash env: KERNEL_TOPICS_PR: ${{ inputs.kernel-topics-pr }} + KERNEL_DIR: ${{ steps.clone.outputs.kernel_dir }} run: | cd "$KERNEL_DIR" echo "Applying kernel-topics PR patch(es): $KERNEL_TOPICS_PR" @@ -275,6 +338,15 @@ runs: KERNEL_CONFIG_INPUT: ${{ inputs.kernel-config }} DKMS_INPUT: ${{ inputs.dkms }} DEBUG_BUILD_INPUT: ${{ inputs.debug-build }} + KERNEL_DIR: ${{ steps.clone.outputs.kernel_dir }} + DISTRO: ${{ steps.env.outputs.distro }} + DOCKER_DISTRO: ${{ steps.env.outputs.docker_distro }} + LOCALVERSION: ${{ steps.localversion.outputs.localversion }} + SNAPSHOT: ${{ steps.localversion.outputs.snapshot }} + DEBIAN_REVISION: ${{ steps.revision.outputs.debian_revision }} + KERNEL_URL: ${{ inputs.kernel-url }} + KERNEL_REF: ${{ steps.ref.outputs.kernel_ref }} + KERNEL_SHA: ${{ steps.clone.outputs.kernel_sha }} run: | # No -x here: keep xtrace off for this step as it assembles the # prepare-source.sh argument list which may include sensitive paths. @@ -288,7 +360,17 @@ runs: --debian-revision "$DEBIAN_REVISION" ) + # The version fields all come from derive-localversion.sh, which + # composed them from the ref and HEAD together. Nothing here appends to + # them: the SHA is already inside LOCALVERSION, and SNAPSHOT and the + # SHA are passed separately for the Debian version, which joins them + # differently. [[ -n "$LOCALVERSION" ]] && ARGS+=(--localversion "$LOCALVERSION") + [[ -n "$SNAPSHOT" ]] && ARGS+=(--snapshot "$SNAPSHOT") + # Full SHA: prepare-source.sh takes the first 12 for the version + # strings and records the whole thing in the changelog, alongside the + # repository and ref that the version strings do not name. + ARGS+=(--git-sha "$KERNEL_SHA" --git-clone "$KERNEL_URL" --git-ref "$KERNEL_REF") [[ -n "$KVER_EXTRA_INPUT" ]] && ARGS+=(--kver-extra "$KVER_EXTRA_INPUT") [[ -n "$KERNEL_CONFIG_INPUT" ]] && ARGS+=(--kernel-config "$KERNEL_CONFIG_INPUT") # Always passed, unlike the optional inputs above: an empty list is a @@ -303,6 +385,8 @@ runs: - name: Pack prepared source tree shell: bash + env: + KERNEL_DIR: ${{ steps.clone.outputs.kernel_dir }} run: | set -euo pipefail tar czf /tmp/kernel-srcpkg.tar.gz \ diff --git a/.github/workflows/build-kernel-debian.yml b/.github/workflows/build-kernel-debian.yml index a885bcea..ead20aad 100644 --- a/.github/workflows/build-kernel-debian.yml +++ b/.github/workflows/build-kernel-debian.yml @@ -203,6 +203,8 @@ jobs: # was dispatched from or the one the calling workflow runs at. - name: Checkout pkg-linux-qcom uses: actions/checkout@v4 + with: + persist-credentials: false - name: Prepare kernel source uses: ./.github/actions/prepare-kernel-source @@ -247,6 +249,8 @@ jobs: # comes from the prepared-source artifact, not from this checkout. - name: Checkout pkg-linux-qcom uses: actions/checkout@v4 + with: + persist-credentials: false - name: Build in Debusine id: debusine @@ -278,6 +282,7 @@ jobs: - name: Checkout debusine-action helpers uses: actions/checkout@v5 with: + persist-credentials: false repository: qualcomm-linux/debusine-action ref: main path: debusine-action diff --git a/.github/workflows/build-kernel-ubuntu.yml b/.github/workflows/build-kernel-ubuntu.yml index 1e4a8aa1..94107d93 100644 --- a/.github/workflows/build-kernel-ubuntu.yml +++ b/.github/workflows/build-kernel-ubuntu.yml @@ -185,6 +185,8 @@ jobs: steps: - name: Checkout pkg-linux-qcom uses: actions/checkout@v4 + with: + persist-credentials: false - name: Prepare kernel source uses: ./.github/actions/prepare-kernel-source @@ -223,12 +225,15 @@ jobs: steps: - name: Checkout pkg-linux-qcom uses: actions/checkout@v4 + with: + persist-credentials: false # Pinned to @main while native kernel-build support is still landing # upstream in qualcomm-linux/docker-pkg-build. - name: Checkout docker-pkg-build uses: actions/checkout@v4 with: + persist-credentials: false repository: qualcomm-linux/docker-pkg-build ref: main path: docker-pkg-build diff --git a/.github/workflows/daily.yml b/.github/workflows/daily.yml index f78b97f3..9a2f8533 100644 --- a/.github/workflows/daily.yml +++ b/.github/workflows/daily.yml @@ -63,6 +63,8 @@ jobs: ubuntu: ${{ steps.set-matrix.outputs.ubuntu }} steps: - uses: actions/checkout@v4 + with: + persist-credentials: false - name: Generate build matrix id: set-matrix diff --git a/.github/workflows/release-kernel-debian.yml b/.github/workflows/release-kernel-debian.yml index 759d054b..46ac9836 100644 --- a/.github/workflows/release-kernel-debian.yml +++ b/.github/workflows/release-kernel-debian.yml @@ -110,6 +110,8 @@ jobs: steps: - name: Checkout pkg-linux-qcom uses: actions/checkout@v4 + with: + persist-credentials: false - name: Prepare kernel source uses: ./.github/actions/prepare-kernel-source @@ -150,6 +152,8 @@ jobs: steps: - name: Checkout pkg-linux-qcom uses: actions/checkout@v4 + with: + persist-credentials: false - name: Build in Debusine id: debusine From d971ac6660462963940d744f0fd3b1b1ef04b1f6 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 12/59] ci: name the Ubuntu build job for where it builds "Build kernel package" said what every build job in the repository does, so next to "Build (Debusine)" it read as the generic one rather than as the other half of a pair. Both jobs build a kernel package; what separates them is where. Name it "Build (Docker)", which is what the rest of the repository already calls this path -- build-kernel.sh --build-mode docker, the README's Docker path -- and the two now differ by the word that actually differs: qcom-next-forky / Build (Debusine) qcom-next-resolute / Build (Docker) The step inside it keeps the name, as the Debusine steps do; it is the job listing that had the ambiguity. Signed-off-by: Christopher Obbard --- .github/workflows/build-kernel-ubuntu.yml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/.github/workflows/build-kernel-ubuntu.yml b/.github/workflows/build-kernel-ubuntu.yml index 94107d93..0856befa 100644 --- a/.github/workflows/build-kernel-ubuntu.yml +++ b/.github/workflows/build-kernel-ubuntu.yml @@ -212,9 +212,13 @@ jobs: # --------------------------------------------------------------------------- # build: build the prepared source in a suite-matched container, then publish. + # + # Named for where the build happens, as the Debian workflows' Build (Debusine) + # is: this one runs in a suite-matched pkg-builder container on the runner + # rather than being submitted to a build service. # --------------------------------------------------------------------------- build: - name: Build kernel package + name: Build (Docker) needs: prepare runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] env: From cd706cb67df5e6e381d210c80ebade8a7fbd4927 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 13/59] ci: pull the builder image instead of rebuilding it every job Every job ran docker_deb_build.py --rebuild, which deletes any existing pkg-builder image and builds a fresh one from docker-pkg-build's Dockerfile. The name it tags looks like a registry reference, ghcr.io/qualcomm-linux/ pkg-builder:, but nothing here ever pushed or pulled it: the docker run that follows was using a local image the previous step had just made. That image is published. docker-pkg-build's container-build-and-upload.yml builds trixie, noble and resolute, validates each by building a test package in it, and pushes them on every push to main and again weekly. So each of our jobs was rebuilding, from the same Dockerfile, an image that upstream had already built and tested -- once for a Debian leg, twice for an Ubuntu one, on ephemeral runners that keep no layer cache between them. Pull trixie and resolute, which covers all eleven daily legs: the ten Debian ones use the trixie image, and resolute is the only Ubuntu one. The package is not public, so the pull logs in with GITHUB_TOKEN under the packages: read permission these workflows already declared and had no use for until now. Suites with no published image still build one on the runner, so the path stays. In prepare-kernel-source both that build and the docker-pkg-build checkout it needs are now conditional, and the Ubuntu build job needs no step at all for them: docker_deb_build.py builds the image itself when it finds it missing, so not pulling is instruction enough. forky is worth a note, having no image of its own in the registry and no Dockerfile in docker-pkg-build to build one from. It uses the trixie image and therefore pulls it, which costs nothing today: Debusine performs the actual suite-specific build, and this container only runs prepare-source.sh, which generates packaging rather than compiling anything. The comment says so, so that the day a forky image appears it is clear what to change. Signed-off-by: Christopher Obbard --- .../actions/prepare-kernel-source/action.yml | 57 ++++++++++++++++--- .github/workflows/build-kernel-ubuntu.yml | 22 ++++++- 2 files changed, 69 insertions(+), 10 deletions(-) diff --git a/.github/actions/prepare-kernel-source/action.yml b/.github/actions/prepare-kernel-source/action.yml index bfe94518..4996b387 100644 --- a/.github/actions/prepare-kernel-source/action.yml +++ b/.github/actions/prepare-kernel-source/action.yml @@ -27,6 +27,13 @@ inputs: suite: description: 'Target suite' required: true + registry-token: + description: > + Token used to pull the pkg-builder image from ghcr.io, which is not a + public package. Defaults to the job's own GITHUB_TOKEN, which needs the + packages: read permission the calling workflows declare. The secrets + context does not reach a composite action, hence an input. + default: ${{ github.token }} expect-family: description: > Build family the calling workflow builds, debian or ubuntu. The suite is @@ -91,13 +98,17 @@ runs: run: | set -euo pipefail # Family and docker image in one place, because they answer the same - # question about a suite. Debian-family suites build in the trixie - # image: docker-pkg-build and pkg-builder only support trixie for - # Debian, and Debusine handles the actual suite-specific build - # internally. Ubuntu-family suites build in an image of their own - # suite. The family list here is the one in + # question about a suite. The family list here is the one in # ci/scripts/resolve-matrix.py, which routes an entry to the workflow # that calls this action. + # + # Ubuntu-family suites use an image of their own suite. Debian-family + # suites all use the trixie image: the registry publishes no forky + # image, and docker-pkg-build has no Dockerfile to build one from + # either, so forky borrows trixie's until one exists. That costs + # nothing today because Debusine performs the actual suite-specific + # build; this container only has to run prepare-source.sh, which + # generates packaging rather than compiling anything. case "$SUITE_INPUT" in trixie|bookworm) FAMILY=debian; DOCKER_DISTRO="$SUITE_INPUT" ;; forky|sid|unstable) FAMILY=debian; DOCKER_DISTRO=trixie ;; @@ -110,11 +121,41 @@ runs: exit 1 fi + # Pull the images qualcomm-linux/docker-pkg-build publishes, and build + # the rest here. Its container-build-and-upload.yml pushes trixie and + # resolute (and noble) on every push to main and weekly, having + # validated each with a test package build, so rebuilding them from the + # same Dockerfile on every job repeats work already done. Anything it + # does not publish still has to be built on the runner. + case "$DOCKER_DISTRO" in + trixie|resolute) IMAGE_SOURCE=pull ;; + *) IMAGE_SOURCE=build ;; + esac + echo "distro=$SUITE_INPUT" >> "$GITHUB_OUTPUT" echo "docker_distro=$DOCKER_DISTRO" >> "$GITHUB_OUTPUT" - echo "Suite $SUITE_INPUT is $FAMILY-family and builds in the $DOCKER_DISTRO image" + echo "image_source=$IMAGE_SOURCE" >> "$GITHUB_OUTPUT" + echo "Suite $SUITE_INPUT is $FAMILY-family and uses the $DOCKER_DISTRO image ($IMAGE_SOURCE)" + - name: Pull builder image + if: ${{ steps.env.outputs.image_source == 'pull' }} + shell: bash + env: + DOCKER_DISTRO: ${{ steps.env.outputs.docker_distro }} + REGISTRY_TOKEN: ${{ inputs.registry-token }} + run: | + # No -x: REGISTRY_TOKEN is in env. + set -euo pipefail + # ghcr.io/qualcomm-linux/pkg-builder is not a public package, so the + # pull needs a login even though the image is only ever read. + echo "$REGISTRY_TOKEN" | docker login ghcr.io -u "$GITHUB_ACTOR" --password-stdin + docker pull "ghcr.io/qualcomm-linux/pkg-builder:$DOCKER_DISTRO" + docker image ls + + # Only for a suite with no published image: the pull path needs neither + # this checkout nor the build below. - name: Checkout docker-pkg-build + if: ${{ steps.env.outputs.image_source == 'build' }} uses: actions/checkout@v4 with: persist-credentials: false @@ -122,11 +163,13 @@ runs: ref: main path: docker-pkg-build - - name: Build docker image + - name: Build builder image + if: ${{ steps.env.outputs.image_source == 'build' }} shell: bash env: DOCKER_DISTRO: ${{ steps.env.outputs.docker_distro }} run: | + set -euo pipefail ./docker-pkg-build/docker_deb_build.py --rebuild -d "$DOCKER_DISTRO" docker image ls diff --git a/.github/workflows/build-kernel-ubuntu.yml b/.github/workflows/build-kernel-ubuntu.yml index 0856befa..33590752 100644 --- a/.github/workflows/build-kernel-ubuntu.yml +++ b/.github/workflows/build-kernel-ubuntu.yml @@ -242,10 +242,26 @@ jobs: ref: main path: docker-pkg-build - - name: Build docker image + - name: Fetch builder image + env: + REGISTRY_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: | - # --rebuild forces a fresh suite-matched pkg-builder image. - ./docker-pkg-build/docker_deb_build.py --rebuild -d "$DISTRO" + # No -x: REGISTRY_TOKEN is in env. + set -euo pipefail + # Same split as prepare-kernel-source: pull what docker-pkg-build + # publishes, build what it does not. Nothing is forced here, because + # docker_deb_build.py builds the image itself when the build below + # finds it missing -- so an unpublished suite needs no step of its + # own, only the absence of a pull. + case "$DISTRO" in + resolute) + echo "$REGISTRY_TOKEN" | docker login ghcr.io -u "$GITHUB_ACTOR" --password-stdin + docker pull "ghcr.io/qualcomm-linux/pkg-builder:$DISTRO" + ;; + *) + echo "No published pkg-builder image for $DISTRO; it will be built on demand." + ;; + esac docker image ls - name: Download prepared source tree From 277a0682479776fa921165546acf610d72994864 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 14/59] ci: name the jobs for what they produce "Prepare kernel source" and "Build (Debusine)" named a stage and a place, so the pair read as different kinds of thing and neither said what came out of it. Name all three for their product instead, and the sequence states itself in the checks list: qcom-next-forky / Generate source package / Build package (Debusine) / Publish to S3 qcom-next-resolute / Generate source package / Build package (Docker) The parenthesis keeps doing what it did before: source package generation is one job whichever family runs it, and only the binary build differs, so that is the only name that has to say where it happens. Display names only. The job ids are untouched, so needs:, the comments and the README all still refer to prepare, build, publish and release as they did. Signed-off-by: Christopher Obbard --- .github/workflows/build-kernel-debian.yml | 4 ++-- .github/workflows/build-kernel-ubuntu.yml | 10 +++++----- .github/workflows/release-kernel-debian.yml | 4 ++-- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/.github/workflows/build-kernel-debian.yml b/.github/workflows/build-kernel-debian.yml index ead20aad..57b2399f 100644 --- a/.github/workflows/build-kernel-debian.yml +++ b/.github/workflows/build-kernel-debian.yml @@ -192,7 +192,7 @@ jobs: # prepare: clone kernel source, inject packaging, run prepare-source.sh. # --------------------------------------------------------------------------- prepare: - name: Prepare kernel source + name: Generate source package runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] steps: # debian/ and ci/ live on the same branch, so one checkout supplies both @@ -232,7 +232,7 @@ jobs: # build: generate the Debian source package and submit it to Debusine. # --------------------------------------------------------------------------- build: - name: Build (Debusine) + name: Build package (Debusine) needs: prepare runs-on: ubuntu-latest environment: Staging diff --git a/.github/workflows/build-kernel-ubuntu.yml b/.github/workflows/build-kernel-ubuntu.yml index 33590752..7bd24b0a 100644 --- a/.github/workflows/build-kernel-ubuntu.yml +++ b/.github/workflows/build-kernel-ubuntu.yml @@ -180,7 +180,7 @@ jobs: # prepare: clone kernel source, inject packaging, run prepare-source.sh. # --------------------------------------------------------------------------- prepare: - name: Prepare kernel source + name: Generate source package runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] steps: - name: Checkout pkg-linux-qcom @@ -213,12 +213,12 @@ jobs: # --------------------------------------------------------------------------- # build: build the prepared source in a suite-matched container, then publish. # - # Named for where the build happens, as the Debian workflows' Build (Debusine) - # is: this one runs in a suite-matched pkg-builder container on the runner - # rather than being submitted to a build service. + # Named for where the build happens, as the Debian workflows' Build package + # (Debusine) is: this one runs in a suite-matched pkg-builder container on + # the runner rather than being submitted to a build service. # --------------------------------------------------------------------------- build: - name: Build (Docker) + name: Build package (Docker) needs: prepare runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] env: diff --git a/.github/workflows/release-kernel-debian.yml b/.github/workflows/release-kernel-debian.yml index 46ac9836..abe8d060 100644 --- a/.github/workflows/release-kernel-debian.yml +++ b/.github/workflows/release-kernel-debian.yml @@ -105,7 +105,7 @@ jobs: # prepare: clone kernel source, inject packaging, run prepare-source.sh. # --------------------------------------------------------------------------- prepare: - name: Prepare kernel source + name: Generate source package runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] steps: - name: Checkout pkg-linux-qcom @@ -136,7 +136,7 @@ jobs: # build: generate the Debian source package and submit it to Debusine. # --------------------------------------------------------------------------- build: - name: Build (Debusine) + name: Build package (Debusine) needs: prepare runs-on: ubuntu-latest environment: Production From 9f62850c8f096baa7c1864f9c933b26c21c8635b Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 15/59] ci: drop the questing and bookworm suites from CI Both were listed as buildable and neither was. questing is skipped by docker_deb_build.py's SKIP_REBUILD_DISTROS and has no published image, so a questing run would build nothing and then fail pulling one. bookworm has no Dockerfile in docker-pkg-build at all, and no published image either. Neither appears in ci/build-matrix.yaml, so both failures were only ever waiting for someone to dispatch them by hand -- which build-kernel-ubuntu.yml invited by naming questing in its suite description. bookworm goes further than the description: it was in resolve-matrix.py's DEBIAN_SUITES, which decides which workflow builds an entry, and it was the only Debian suite mapping to an image of its own name. Without it every Debian-family suite maps to trixie, so the two case arms that did that become one arm saying so. The packaging keeps questing. prepare-source.sh accepts it in VALID_DISTROS and debian/README.md lists it as a supported distribution, which is a claim about the packaging rather than about CI, and it is true: what is missing is a container image to build it in, not support for the suite. bookworm was never in either, so nothing there to remove. Signed-off-by: Christopher Obbard --- .../actions/prepare-kernel-source/action.yml | 20 +++++++++---------- .github/workflows/build-kernel-ubuntu.yml | 4 ++-- ci/scripts/resolve-matrix.py | 2 +- 3 files changed, 13 insertions(+), 13 deletions(-) diff --git a/.github/actions/prepare-kernel-source/action.yml b/.github/actions/prepare-kernel-source/action.yml index 4996b387..d109e5a5 100644 --- a/.github/actions/prepare-kernel-source/action.yml +++ b/.github/actions/prepare-kernel-source/action.yml @@ -102,17 +102,17 @@ runs: # ci/scripts/resolve-matrix.py, which routes an entry to the workflow # that calls this action. # - # Ubuntu-family suites use an image of their own suite. Debian-family - # suites all use the trixie image: the registry publishes no forky - # image, and docker-pkg-build has no Dockerfile to build one from - # either, so forky borrows trixie's until one exists. That costs - # nothing today because Debusine performs the actual suite-specific - # build; this container only has to run prepare-source.sh, which - # generates packaging rather than compiling anything. + # Ubuntu-family suites use an image of their own suite. Every + # Debian-family suite uses the trixie image, forky and sid included: + # the registry publishes no image for either, and docker-pkg-build has + # no Dockerfile to build one from, so they borrow trixie's until one + # exists. That costs nothing today because Debusine performs the actual + # suite-specific build; this container only has to run + # prepare-source.sh, which generates packaging rather than compiling + # anything. case "$SUITE_INPUT" in - trixie|bookworm) FAMILY=debian; DOCKER_DISTRO="$SUITE_INPUT" ;; - forky|sid|unstable) FAMILY=debian; DOCKER_DISTRO=trixie ;; - *) FAMILY=ubuntu; DOCKER_DISTRO="$SUITE_INPUT" ;; + trixie|forky|sid|unstable) FAMILY=debian; DOCKER_DISTRO=trixie ;; + *) FAMILY=ubuntu; DOCKER_DISTRO="$SUITE_INPUT" ;; esac if [[ "$FAMILY" != "$EXPECT_FAMILY" ]]; then diff --git a/.github/workflows/build-kernel-ubuntu.yml b/.github/workflows/build-kernel-ubuntu.yml index 7bd24b0a..1499a35d 100644 --- a/.github/workflows/build-kernel-ubuntu.yml +++ b/.github/workflows/build-kernel-ubuntu.yml @@ -28,7 +28,7 @@ on: type: string default: 'qcom-next' suite: - description: 'Target Ubuntu suite (noble, questing, resolute)' + description: 'Target Ubuntu suite (noble, resolute)' type: string default: 'resolute' kernel-branch: @@ -102,7 +102,7 @@ on: default: 'qcom-next' type: string suite: - description: 'Target Ubuntu suite (noble, questing, resolute). A Debian suite here is rejected before anything is built; dispatch build-kernel-debian for those.' + description: 'Target Ubuntu suite (noble, resolute). A Debian suite here is rejected before anything is built; dispatch build-kernel-debian for those.' required: true default: 'resolute' type: string diff --git a/ci/scripts/resolve-matrix.py b/ci/scripts/resolve-matrix.py index 8151cfdf..edadb6a9 100755 --- a/ci/scripts/resolve-matrix.py +++ b/ci/scripts/resolve-matrix.py @@ -93,7 +93,7 @@ # runners, containers and publish steps, so a caller selects entries by family # and calls the one workflow that builds them -- rather than every build leg # starting both and skipping one. -DEBIAN_SUITES = ("trixie", "forky", "sid", "unstable", "bookworm") +DEBIAN_SUITES = ("trixie", "forky", "sid", "unstable") FAMILIES = ("debian", "ubuntu") From 1150173fd6796e935608959b8b42f163642a8424 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 16/59] ci: pin every third-party action to the commit its tag names Every action this repository uses was floating on a mutable ref. Most named a major-version tag, two checkouts were already pinned to a commit while thirteen others were not, and the legacy build-kernel.yml named `aws`, a branch, so it ran whatever had last landed there -- while publishing artifacts to S3. A tag can be repointed at any commit and a branch moves by design, which is the supply-chain hole pinning closes. Pin every use to the commit the release tag names, with the version in a comment so the next reader knows what is pinned without resolving a SHA: actions/checkout 3d3c42e5aac5ba805825da76410c181273ba90b1 v7.0.1 actions/upload-artifact 043fb46d1a93c77aae656e7c1c64a875d1fc6a0a v7.0.1 actions/download-artifact 3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c v8.0.1 upload-private-artifact-action 4940e9327cd7386acdb908b61c714c079b7d754b aws-v4 Each SHA was checked against the tag it claims to be rather than taken on trust. download-artifact was four majors behind; it is the counterpart of the upload-artifact pinned alongside it, since prepare uploads the prepared source tree and both build paths download it again. v8's inputs are a superset of v4's, adding digest-mismatch and skip-decompress and dropping nothing, so the name and path this repository passes still mean what they meant. upload-private-artifact-action is pinned, not bumped: aws-v5 exists but this stays on aws-v4, so the branch stops deciding what the legacy workflow runs and all three uses agree on one version. Nothing it passes is affected -- s3_bucket, path and destination are inputs of both, and the inputs aws-v5 adds are ones no caller here sets. This raises a runner requirement. checkout, upload-artifact and download-artifact all run on Node.js 24, which needs Actions Runner 2.327.1 or newer, and the prepare and build jobs run on the self-hosted arm64 runners. Those need to be at that version before this merges, or every job that checks out will fail on them. Signed-off-by: Christopher Obbard --- .github/actions/debusine-build/action.yml | 4 ++-- .github/actions/prepare-kernel-source/action.yml | 4 ++-- .github/workflows/build-kernel-debian.yml | 8 ++++---- .github/workflows/build-kernel-ubuntu.yml | 10 +++++----- .github/workflows/daily.yml | 2 +- .github/workflows/pr-build.yml | 2 +- .github/workflows/release-kernel-debian.yml | 6 +++--- .github/workflows/release.yml | 2 +- 8 files changed, 19 insertions(+), 19 deletions(-) diff --git a/.github/actions/debusine-build/action.yml b/.github/actions/debusine-build/action.yml index 6355aea6..e089dff9 100644 --- a/.github/actions/debusine-build/action.yml +++ b/.github/actions/debusine-build/action.yml @@ -64,7 +64,7 @@ runs: using: composite steps: - name: Checkout debusine-action helpers - uses: actions/checkout@v5 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false repository: qualcomm-linux/debusine-action @@ -75,7 +75,7 @@ runs: lib - name: Download prepared source tree - uses: actions/download-artifact@v4 + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 with: name: kernel-srcpkg-${{ inputs.flavour }}-${{ inputs.suite }} path: /tmp/srcpkg-artifact diff --git a/.github/actions/prepare-kernel-source/action.yml b/.github/actions/prepare-kernel-source/action.yml index d109e5a5..1aba18a2 100644 --- a/.github/actions/prepare-kernel-source/action.yml +++ b/.github/actions/prepare-kernel-source/action.yml @@ -156,7 +156,7 @@ runs: # this checkout nor the build below. - name: Checkout docker-pkg-build if: ${{ steps.env.outputs.image_source == 'build' }} - uses: actions/checkout@v4 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false repository: qualcomm-linux/docker-pkg-build @@ -437,7 +437,7 @@ runs: "$(basename "$KERNEL_DIR")" - name: Upload prepared source tree artifact - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: kernel-srcpkg-${{ inputs.flavour }}-${{ inputs.suite }} path: /tmp/kernel-srcpkg.tar.gz diff --git a/.github/workflows/build-kernel-debian.yml b/.github/workflows/build-kernel-debian.yml index 57b2399f..49572f2d 100644 --- a/.github/workflows/build-kernel-debian.yml +++ b/.github/workflows/build-kernel-debian.yml @@ -202,7 +202,7 @@ jobs: # An empty ref leaves actions/checkout on github.sha, the commit this run # was dispatched from or the one the calling workflow runs at. - name: Checkout pkg-linux-qcom - uses: actions/checkout@v4 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false @@ -248,7 +248,7 @@ jobs: # Only for the debusine-build action itself: the package being built # comes from the prepared-source artifact, not from this checkout. - name: Checkout pkg-linux-qcom - uses: actions/checkout@v4 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false @@ -280,7 +280,7 @@ jobs: shell: bash steps: - name: Checkout debusine-action helpers - uses: actions/checkout@v5 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false repository: qualcomm-linux/debusine-action @@ -326,7 +326,7 @@ jobs: # Keep flavour and suite in the destination so concurrent matrix legs # cannot overwrite or mix package outputs. Consumers must select the # flavour and suite they intend to install. - uses: qualcomm-linux/upload-private-artifact-action@aws-v4 + uses: qualcomm-linux/upload-private-artifact-action@4940e9327cd7386acdb908b61c714c079b7d754b # aws-v4 with: s3_bucket: ${{ vars.ARTIFACT_S3_BUCKET }} path: deb-artifacts diff --git a/.github/workflows/build-kernel-ubuntu.yml b/.github/workflows/build-kernel-ubuntu.yml index 1499a35d..d0c2d3f2 100644 --- a/.github/workflows/build-kernel-ubuntu.yml +++ b/.github/workflows/build-kernel-ubuntu.yml @@ -184,7 +184,7 @@ jobs: runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] steps: - name: Checkout pkg-linux-qcom - uses: actions/checkout@v4 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false @@ -228,14 +228,14 @@ jobs: shell: bash steps: - name: Checkout pkg-linux-qcom - uses: actions/checkout@v4 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false # Pinned to @main while native kernel-build support is still landing # upstream in qualcomm-linux/docker-pkg-build. - name: Checkout docker-pkg-build - uses: actions/checkout@v4 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false repository: qualcomm-linux/docker-pkg-build @@ -265,7 +265,7 @@ jobs: docker image ls - name: Download prepared source tree - uses: actions/download-artifact@v4 + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 with: name: kernel-srcpkg-${{ inputs.flavour }}-${{ inputs.suite }} path: /tmp/srcpkg-artifact @@ -299,7 +299,7 @@ jobs: # Keep flavour and suite in the destination so concurrent matrix legs # cannot overwrite or mix package outputs. Consumers must select the # flavour and suite they intend to install. - uses: qualcomm-linux/upload-private-artifact-action@aws-v4 + uses: qualcomm-linux/upload-private-artifact-action@4940e9327cd7386acdb908b61c714c079b7d754b # aws-v4 with: s3_bucket: ${{ vars.ARTIFACT_S3_BUCKET }} path: kernel-build/${{ inputs.suite }} diff --git a/.github/workflows/daily.yml b/.github/workflows/daily.yml index 9a2f8533..ef184b80 100644 --- a/.github/workflows/daily.yml +++ b/.github/workflows/daily.yml @@ -62,7 +62,7 @@ jobs: debian: ${{ steps.set-matrix.outputs.debian }} ubuntu: ${{ steps.set-matrix.outputs.ubuntu }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false diff --git a/.github/workflows/pr-build.yml b/.github/workflows/pr-build.yml index 7b3f699a..ed8f0c71 100644 --- a/.github/workflows/pr-build.yml +++ b/.github/workflows/pr-build.yml @@ -33,7 +33,7 @@ jobs: debian: ${{ steps.set-matrix.outputs.debian }} ubuntu: ${{ steps.set-matrix.outputs.ubuntu }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ github.event.pull_request.head.sha }} persist-credentials: false diff --git a/.github/workflows/release-kernel-debian.yml b/.github/workflows/release-kernel-debian.yml index abe8d060..3b1ad19d 100644 --- a/.github/workflows/release-kernel-debian.yml +++ b/.github/workflows/release-kernel-debian.yml @@ -109,7 +109,7 @@ jobs: runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] steps: - name: Checkout pkg-linux-qcom - uses: actions/checkout@v4 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false @@ -151,7 +151,7 @@ jobs: srcpkg_version: ${{ steps.debusine.outputs.srcpkg_version }} steps: - name: Checkout pkg-linux-qcom - uses: actions/checkout@v4 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false @@ -189,7 +189,7 @@ jobs: shell: bash steps: - name: Checkout debusine-action helpers - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false repository: qualcomm-linux/debusine-action diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index bdbbb1e1..b31fc582 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -51,7 +51,7 @@ jobs: outputs: matrix: ${{ steps.set-matrix.outputs.matrix }} steps: - - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false From 51435a41c429af4d15b2b6bca8034dccbbadf44d Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 17/59] ci: scope the token per job and say what each permission is for Every workflow granted contents: read and packages: read to all of its jobs, so a job that only reads the repository still held a token that could read packages, and nothing recorded which job needed which. Give each job the permissions it uses and name the consumer in a comment, so the next person to add a step can tell whether the token already covers it: permissions: contents: read # actions/checkout packages: read # docker pull ghcr.io/qualcomm-linux/pkg-builder Publish to S3 is the one job that narrows: it checks out debusine-action and reads the built packages out of Debusine over HTTP, and pulls no image, so it keeps contents: read alone. Every other job either pulls the pkg-builder image or runs in the debusine-pkg-builder container, and the comment says which. Job-level permissions replace the workflow-level ones rather than adding to them, so each block lists everything its job needs. On the jobs that call a reusable workflow the block is a ceiling for the whole call, which is why daily.yml, pr-build.yml and release.yml grant packages: read there: the workflow they call pulls images even though the calling job runs nothing itself. Signed-off-by: Christopher Obbard --- .github/workflows/build-kernel-debian.yml | 8 ++++++++ .github/workflows/build-kernel-ubuntu.yml | 6 ++++++ .github/workflows/daily.yml | 8 ++++++++ .github/workflows/pr-build.yml | 8 ++++++++ .github/workflows/release-kernel-debian.yml | 9 +++++++++ .github/workflows/release.yml | 5 +++++ 6 files changed, 44 insertions(+) diff --git a/.github/workflows/build-kernel-debian.yml b/.github/workflows/build-kernel-debian.yml index 49572f2d..e6dcf24d 100644 --- a/.github/workflows/build-kernel-debian.yml +++ b/.github/workflows/build-kernel-debian.yml @@ -194,6 +194,9 @@ jobs: prepare: name: Generate source package runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] + permissions: + contents: read # actions/checkout + packages: read # docker pull ghcr.io/qualcomm-linux/pkg-builder steps: # debian/ and ci/ live on the same branch, so one checkout supplies both # the packaging and the scripts, the delivery matrix they read, and the @@ -235,6 +238,9 @@ jobs: name: Build package (Debusine) needs: prepare runs-on: ubuntu-latest + permissions: + contents: read # actions/checkout + packages: read # container: ghcr.io/qualcomm-linux/debusine-pkg-builder environment: Staging container: image: ghcr.io/qualcomm-linux/debusine-pkg-builder:trixie @@ -275,6 +281,8 @@ jobs: name: Publish to S3 needs: build runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] + permissions: + contents: read # actions/checkout defaults: run: shell: bash diff --git a/.github/workflows/build-kernel-ubuntu.yml b/.github/workflows/build-kernel-ubuntu.yml index d0c2d3f2..270c15b3 100644 --- a/.github/workflows/build-kernel-ubuntu.yml +++ b/.github/workflows/build-kernel-ubuntu.yml @@ -182,6 +182,9 @@ jobs: prepare: name: Generate source package runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] + permissions: + contents: read # actions/checkout + packages: read # docker pull ghcr.io/qualcomm-linux/pkg-builder steps: - name: Checkout pkg-linux-qcom uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -221,6 +224,9 @@ jobs: name: Build package (Docker) needs: prepare runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] + permissions: + contents: read # actions/checkout + packages: read # docker pull ghcr.io/qualcomm-linux/pkg-builder env: DISTRO: ${{ inputs.suite }} defaults: diff --git a/.github/workflows/daily.yml b/.github/workflows/daily.yml index ef184b80..12eda419 100644 --- a/.github/workflows/daily.yml +++ b/.github/workflows/daily.yml @@ -58,6 +58,8 @@ jobs: configure-matrix: name: Configure build matrix runs-on: ubuntu-latest + permissions: + contents: read # actions/checkout outputs: debian: ${{ steps.set-matrix.outputs.debian }} ubuntu: ${{ steps.set-matrix.outputs.ubuntu }} @@ -137,6 +139,9 @@ jobs: fail-fast: false matrix: include: ${{ fromJson(needs.configure-matrix.outputs.debian) }} + permissions: + contents: read # actions/checkout in the called workflow + packages: read # ghcr.io images it pulls uses: ./.github/workflows/build-kernel-debian.yml with: suite: ${{ matrix.suite }} @@ -166,6 +171,9 @@ jobs: fail-fast: false matrix: include: ${{ fromJson(needs.configure-matrix.outputs.ubuntu) }} + permissions: + contents: read # actions/checkout in the called workflow + packages: read # ghcr.io images it pulls uses: ./.github/workflows/build-kernel-ubuntu.yml with: suite: ${{ matrix.suite }} diff --git a/.github/workflows/pr-build.yml b/.github/workflows/pr-build.yml index ed8f0c71..8dfeea50 100644 --- a/.github/workflows/pr-build.yml +++ b/.github/workflows/pr-build.yml @@ -29,6 +29,8 @@ jobs: configure-matrix: name: Configure build matrix runs-on: ubuntu-latest + permissions: + contents: read # actions/checkout outputs: debian: ${{ steps.set-matrix.outputs.debian }} ubuntu: ${{ steps.set-matrix.outputs.ubuntu }} @@ -74,6 +76,9 @@ jobs: fail-fast: false matrix: include: ${{ fromJson(needs.configure-matrix.outputs.debian) }} + permissions: + contents: read # actions/checkout in the called workflow + packages: read # ghcr.io images it pulls uses: ./.github/workflows/build-kernel-debian.yml with: suite: ${{ matrix.suite }} @@ -104,6 +109,9 @@ jobs: fail-fast: false matrix: include: ${{ fromJson(needs.configure-matrix.outputs.ubuntu) }} + permissions: + contents: read # actions/checkout in the called workflow + packages: read # ghcr.io images it pulls uses: ./.github/workflows/build-kernel-ubuntu.yml with: suite: ${{ matrix.suite }} diff --git a/.github/workflows/release-kernel-debian.yml b/.github/workflows/release-kernel-debian.yml index 3b1ad19d..f4eeb40a 100644 --- a/.github/workflows/release-kernel-debian.yml +++ b/.github/workflows/release-kernel-debian.yml @@ -107,6 +107,9 @@ jobs: prepare: name: Generate source package runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] + permissions: + contents: read # actions/checkout + packages: read # docker pull ghcr.io/qualcomm-linux/pkg-builder steps: - name: Checkout pkg-linux-qcom uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -139,6 +142,9 @@ jobs: name: Build package (Debusine) needs: prepare runs-on: ubuntu-latest + permissions: + contents: read # actions/checkout + packages: read # container: ghcr.io/qualcomm-linux/debusine-pkg-builder environment: Production container: image: ghcr.io/qualcomm-linux/debusine-pkg-builder:trixie @@ -177,6 +183,9 @@ jobs: name: Release to Debusine needs: build runs-on: ubuntu-latest + permissions: + contents: read # actions/checkout + packages: read # container: ghcr.io/qualcomm-linux/debusine-pkg-builder environment: Production container: image: ghcr.io/qualcomm-linux/debusine-pkg-builder:trixie diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index b31fc582..a96f4caa 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -48,6 +48,8 @@ jobs: configure-matrix: name: Configure release matrix runs-on: ubuntu-latest + permissions: + contents: read # actions/checkout outputs: matrix: ${{ steps.set-matrix.outputs.matrix }} steps: @@ -105,6 +107,9 @@ jobs: fail-fast: false matrix: include: ${{ fromJson(needs.configure-matrix.outputs.matrix) }} + permissions: + contents: read # actions/checkout in the called workflow + packages: read # ghcr.io images it pulls uses: ./.github/workflows/release-kernel-debian.yml with: suite: ${{ matrix.suite }} From 253e091a09ab1befe9b6f1cb79cdb8414927b276 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 18/59] ci: dispatch builds by name, defaulting to the whole matrix Every manual build already had to name something the matrix defines, but each workflow asked for it differently, and the two build workflows asked for things the matrix had long since taken over. daily.yml offered a Build scope dropdown whose three options -- full matrix, named builds, one flavour -- were three ways of writing a set of entries, one of which needed a second input to say which set. release.yml offered the same idea under a different name and a different default. build-kernel-debian.yml and build-kernel-ubuntu.yml offered neither: their dispatch predates one entry being one package, and still asked a person to type git-clone, srcpkg, binpkg, kernel-config, dkms and a Debian revision, with defaults frozen at whatever qcom-next/trixie looked like when the form was written. Dispatching one of those built whatever the form said, not what the matrix says, and nothing reconciled the two. So there is now one input, in the two workflows that resolve a matrix: builds, defaulting to all, otherwise a comma-separated list of names. A scheduled run carries no inputs and lands on the same default, which is what it wanted anyway. Selecting a flavour across its suites is naming its entries; the --flavour filter stays in resolve-matrix.py for local use. A name matching no entry of that delivery type still fails the run with the names that do, so a typo cannot quietly narrow a release. daily.yml is the manual entry point that build-kernel-* stops being. It resolves the selection into both families, so one dispatch can name Debian and Ubuntu builds together and each is called by the workflow that builds it -- which is the thing a dispatch of a single-family workflow could never do. The build workflows are workflow_call only now: one run of either is one matrix entry, and a reusable workflow cannot call itself to fan out over a list. The three inputs they offered that the matrix deliberately does not carry -- debug-build, qcom-next-pr, kernel-topics-pr -- move to daily.yml, where they apply to every selected build. They describe a one-off validation run rather than a delivery target, which is why they are inputs and not entry fields. The rest of what that form asked for is gone: it is in the entry, and typing it again was the way to disagree with it. resolve-matrix.py now rejects a build named all, which the dispatch form would otherwise read as every entry, leaving that build unreachable by name. Signed-off-by: Christopher Obbard --- .../actions/prepare-kernel-source/action.yml | 12 +- .github/workflows/build-kernel-debian.yml | 90 +------------ .github/workflows/build-kernel-ubuntu.yml | 89 +------------ .github/workflows/daily.yml | 95 +++++++------- .github/workflows/release.yml | 50 +++---- README.md | 122 +++++++++--------- ci/build-matrix.yaml | 3 +- ci/scripts/resolve-matrix.py | 12 ++ 8 files changed, 157 insertions(+), 316 deletions(-) diff --git a/.github/actions/prepare-kernel-source/action.yml b/.github/actions/prepare-kernel-source/action.yml index 1aba18a2..9a993ad2 100644 --- a/.github/actions/prepare-kernel-source/action.yml +++ b/.github/actions/prepare-kernel-source/action.yml @@ -292,12 +292,12 @@ runs: fi - name: Resolve Debian revision - # Callers driven by the matrix pass the entry's own debian-revision - # straight through. A direct workflow_dispatch that leaves it empty looks - # the same entry up itself, so a manual build of a configured build gets - # exactly the revision the daily build would have used. The lookup is - # always the Daily entry: direct dispatch is build-only and never - # promotes. + # The build workflows pass the matrix entry's own debian-revision + # straight through, so this lookup is the fallback for a caller that has + # no entry in hand and leaves the input empty: it looks up the Daily + # entry for the build name and suite it was given, and builds at the + # revision the daily build would have used. Always the Daily entry -- + # a release is driven by release.yml, which passes its own revision. id: revision shell: bash env: diff --git a/.github/workflows/build-kernel-debian.yml b/.github/workflows/build-kernel-debian.yml index e6dcf24d..e3e2ca1a 100644 --- a/.github/workflows/build-kernel-debian.yml +++ b/.github/workflows/build-kernel-debian.yml @@ -8,6 +8,12 @@ name: build-kernel-debian # reaches this file has already decided the build belongs on the Debusine # path, so every job in it runs. # +# workflow_call only. One run of this workflow is one matrix entry, and a +# reusable workflow cannot fan itself out over several, so manual builds are +# dispatched from daily.yml -- which resolves the entries and calls this once +# per Debian entry it selected. That also keeps every value describing a build +# in ci/build-matrix.yaml, rather than in a dispatch form that drifts from it. +# # release-kernel-debian.yml is the same build with the release tail: it # promotes to a target Debusine workspace instead of publishing to S3. # @@ -99,90 +105,6 @@ on: description: 'Debusine token for the CI child workspace' required: false - # Ad-hoc build of one Debian suite. Release promotion is not offered here: - # a manual build never promotes, so it always takes the S3 publish path. - workflow_dispatch: - inputs: - build: - description: 'Name of this build, as listed in ci/build-matrix.yaml' - required: true - default: 'qcom-next-trixie' - type: string - flavour: - description: 'Kernel flavour: the LOCALVERSION suffix and so the kernel release identity' - required: true - default: 'qcom-next' - type: string - suite: - description: 'Target Debian suite (trixie, forky, sid). An Ubuntu suite here is rejected before anything is built; dispatch build-kernel-ubuntu for those.' - required: true - default: 'trixie' - type: string - ref-strategy: - description: 'Kernel ref strategy' - required: true - default: 'latest_tag' - type: choice - options: - - latest_tag - - branch_tip - - pinned_ref - kernel-branch: - description: 'Branch for branch_tip or immutable ref for pinned_ref (ignored for latest_tag)' - required: false - default: 'qcom-next' - tag-pattern: - description: 'Tag glob used only by latest_tag (ignored for branch_tip and pinned_ref)' - required: false - default: 'qcom-next-*' - git-clone: - description: 'Advanced: custom kernel repository URL' - required: false - default: 'https://github.com/qualcomm-linux/kernel' - srcpkg: - description: 'Advanced: Debian source package name' - required: false - default: 'linux-qcom-next' - binpkg: - description: 'Advanced: kernel image metapackage name' - required: false - default: 'linux-image-qcom-next' - kernel-config: - description: 'Advanced: extra config fragments applied on top of all of debian/config-available/ (e.g. intree:arch/arm64/configs/qcom_debug.config)' - required: false - default: '' - dkms: - description: 'Advanced: out-of-tree DKMS modules to build and bundle into the kernel image, comma-separated and without the -dkms suffix (e.g. kgsl,camx); empty bundles nothing' - required: false - default: '' - debian-revision: - description: 'Advanced: Debian revision override (empty takes the Daily revision the matrix gives this build and suite)' - required: false - default: '' - localversion: - description: 'Advanced: LOCALVERSION override (auto-derived from the resolved ref if empty)' - required: false - default: '' - kver-extra: - description: 'Advanced: extra package-version suffix (e.g. -ci42)' - required: false - default: '' - debug-build: - description: 'Advanced: enable the debug build configuration' - type: boolean - required: false - default: false - qcom-next-pr: - description: 'Advanced Qualcomm-only override: qcom-next PR numbers to merge (e.g. "42 43")' - type: string - required: false - default: '' - kernel-topics-pr: - description: 'Advanced Qualcomm-only override: kernel-topics PR numbers to apply as patches' - type: string - required: false - default: '' - permissions: contents: read packages: read diff --git a/.github/workflows/build-kernel-ubuntu.yml b/.github/workflows/build-kernel-ubuntu.yml index 270c15b3..33a4b49f 100644 --- a/.github/workflows/build-kernel-ubuntu.yml +++ b/.github/workflows/build-kernel-ubuntu.yml @@ -12,6 +12,12 @@ name: build-kernel-ubuntu # entry for an Ubuntu suite is rejected by resolve-matrix.py rather than built # here and quietly not promoted. # +# workflow_call only. One run of this workflow is one matrix entry, and a +# reusable workflow cannot fan itself out over several, so manual builds are +# dispatched from daily.yml -- which resolves the entries and calls this once +# per Ubuntu entry it selected. That also keeps every value describing a build +# in ci/build-matrix.yaml, rather than in a dispatch form that drifts from it. +# # Jobs: # prepare (self-hosted): clone, patch, inject debian/, generate the source. # build (self-hosted): build the .deb files and upload them to S3. @@ -88,89 +94,6 @@ on: type: string default: '' - # Ad-hoc build of one Ubuntu suite. - workflow_dispatch: - inputs: - build: - description: 'Name of this build, as listed in ci/build-matrix.yaml' - required: true - default: 'qcom-next-resolute' - type: string - flavour: - description: 'Kernel flavour: the LOCALVERSION suffix and so the kernel release identity' - required: true - default: 'qcom-next' - type: string - suite: - description: 'Target Ubuntu suite (noble, resolute). A Debian suite here is rejected before anything is built; dispatch build-kernel-debian for those.' - required: true - default: 'resolute' - type: string - ref-strategy: - description: 'Kernel ref strategy' - required: true - default: 'latest_tag' - type: choice - options: - - latest_tag - - branch_tip - - pinned_ref - kernel-branch: - description: 'Branch for branch_tip or immutable ref for pinned_ref (ignored for latest_tag)' - required: false - default: 'qcom-next' - tag-pattern: - description: 'Tag glob used only by latest_tag (ignored for branch_tip and pinned_ref)' - required: false - default: 'qcom-next-*' - git-clone: - description: 'Advanced: custom kernel repository URL' - required: false - default: 'https://github.com/qualcomm-linux/kernel' - srcpkg: - description: 'Advanced: Debian source package name' - required: false - default: 'linux-qcom-next' - binpkg: - description: 'Advanced: kernel image metapackage name' - required: false - default: 'linux-image-qcom-next' - kernel-config: - description: 'Advanced: extra config fragments applied on top of all of debian/config-available/ (e.g. intree:arch/arm64/configs/qcom_debug.config)' - required: false - default: '' - dkms: - description: 'Advanced: out-of-tree DKMS modules to build and bundle into the kernel image, comma-separated and without the -dkms suffix (e.g. kgsl,camx); empty bundles nothing' - required: false - default: '' - debian-revision: - description: 'Advanced: Debian revision override (empty takes the Daily revision the matrix gives this build and suite)' - required: false - default: '' - localversion: - description: 'Advanced: LOCALVERSION override (auto-derived from the resolved ref if empty)' - required: false - default: '' - kver-extra: - description: 'Advanced: extra package-version suffix (e.g. -ci42)' - required: false - default: '' - debug-build: - description: 'Advanced: enable the debug build configuration' - type: boolean - required: false - default: false - qcom-next-pr: - description: 'Advanced Qualcomm-only override: qcom-next PR numbers to merge (e.g. "42 43")' - type: string - required: false - default: '' - kernel-topics-pr: - description: 'Advanced Qualcomm-only override: kernel-topics PR numbers to apply as patches' - type: string - required: false - default: '' - permissions: contents: read packages: read diff --git a/.github/workflows/daily.yml b/.github/workflows/daily.yml index 12eda419..6c06dd8c 100644 --- a/.github/workflows/daily.yml +++ b/.github/workflows/daily.yml @@ -1,9 +1,12 @@ name: daily # Daily kernel package builds. Runs at 3:00 PM PST (23:00 UTC), one parallel -# build per Daily entry in ci/build-matrix.yaml. Manual dispatch can run the -# full Daily matrix, a named set of builds, or every suite of one flavour. -# Build names and flavours are matrix values, so new ones need no workflow edit. +# build per Daily entry in ci/build-matrix.yaml. +# +# This is also the manual build entry point. Everything about a build comes +# from its matrix entry, so a dispatch only says which entries to run: "all", +# or a comma-separated list of build names. Build names are matrix values, so +# a new one needs no workflow edit. # # Matrix schema: ci/build-matrix.yaml # One entry is one generated package, so ci/scripts/resolve-matrix.py selects @@ -22,24 +25,29 @@ on: - cron: '0 23 * * *' workflow_dispatch: inputs: - build-scope: - description: 'Choose the whole matrix, named build(s), or every suite of one flavour' - type: choice - required: true - default: 'Selected builds' - options: - - 'Full matrix' - - 'Selected builds' - - 'Selected flavour (all suites)' builds: - description: 'Comma-separated build names, e.g. qcom-next-trixie,qcom-next-debug-forky (Selected builds)' + description: 'Builds to run: all, or comma-separated Daily build names from ci/build-matrix.yaml (e.g. qcom-next-trixie,qcom-next-debug-forky)' required: true - default: 'qcom-next-trixie' + default: 'all' type: string - flavour: - description: 'Kernel flavour to build in every suite it targets (Selected flavour)' - required: true - default: 'qcom-next' + # The matrix says nothing about these: they are one-off validation + # overrides, so they are inputs rather than entry fields. They apply to + # every selected build, and a scheduled run leaves them at their + # defaults. + debug-build: + description: 'Advanced: enable the debug build configuration' + required: false + default: false + type: boolean + qcom-next-pr: + description: 'Advanced Qualcomm-only override: qcom-next PR numbers to merge (e.g. "42 43")' + required: false + default: '' + type: string + kernel-topics-pr: + description: 'Advanced Qualcomm-only override: kernel-topics PR numbers to apply as patches' + required: false + default: '' type: string permissions: @@ -51,9 +59,9 @@ jobs: # configure-matrix: select the Daily entries from ci/build-matrix.yaml. # # Delegates to ci/scripts/resolve-matrix.py, which validates the whole matrix - # and returns the entries with type: Daily. Manual dispatch selects the - # complete matrix, a comma-separated list of build names, or one flavour - # across every suite it targets. + # and returns the entries with type: Daily. A dispatch narrows that to a + # comma-separated list of build names; "all", and a scheduled run, take the + # whole Daily matrix. # --------------------------------------------------------------------------- configure-matrix: name: Configure build matrix @@ -71,37 +79,24 @@ jobs: - name: Generate build matrix id: set-matrix env: - BUILD_SCOPE_INPUT: ${{ github.event.inputs.build-scope }} - INPUT_BUILDS: ${{ github.event.inputs.builds }} - INPUT_FLAVOUR: ${{ github.event.inputs.flavour }} + INPUT_BUILDS: ${{ inputs.builds }} run: | set -euo pipefail - BUILD_SCOPE="$BUILD_SCOPE_INPUT" + # A scheduled run carries no inputs, so it lands on the dispatch + # default: the whole Daily matrix. + BUILDS="${INPUT_BUILDS:-all}" - if [[ "${{ github.event_name }}" == "schedule" ]]; then - BUILD_SCOPE="Full matrix" + args=(--type Daily) + if [[ "$BUILDS" == "all" ]]; then + echo "Source: ci/build-matrix.yaml (every Daily entry)" + else + echo "Source: ci/build-matrix.yaml (builds: $BUILDS)" + # A name matching no Daily entry fails here, so a typo stops the + # run rather than quietly narrowing it to nothing. + args+=(--build "$BUILDS") fi - case "$BUILD_SCOPE" in - "Full matrix") - echo "Source: ci/build-matrix.yaml (full Daily matrix)" - args=(--type Daily) - ;; - "Selected builds") - echo "Source: ci/build-matrix.yaml (builds: $INPUT_BUILDS)" - args=(--type Daily --build "$INPUT_BUILDS") - ;; - "Selected flavour (all suites)") - echo "Source: ci/build-matrix.yaml (flavour: $INPUT_FLAVOUR, all suites)" - args=(--type Daily --flavour "$INPUT_FLAVOUR") - ;; - *) - echo "ERROR: unsupported build scope: $BUILD_SCOPE" >&2 - exit 1 - ;; - esac - # One selection, split by build path. The two families are built by # different workflows, so each gets the entries it builds and neither # run starts a job for the other. A narrowed dispatch can leave one @@ -159,6 +154,11 @@ jobs: localversion: ${{ matrix.localversion || '' }} kver-extra: ${{ matrix.kver_extra || '' }} debusine-parent-workspace: ${{ matrix.debusine_parent_workspace || '' }} + # Dispatch-only validation overrides. A scheduled run carries no inputs, + # so each falls back to the off value and builds the matrix as written. + debug-build: ${{ inputs.debug-build || false }} + qcom-next-pr: ${{ inputs.qcom-next-pr || '' }} + kernel-topics-pr: ${{ inputs.kernel-topics-pr || '' }} secrets: DEBUSINE_USER: ${{ secrets.DEBUSINE_USER }} DEBUSINE_TOKEN: ${{ secrets.DEBUSINE_TOKEN }} @@ -190,3 +190,6 @@ jobs: debian-revision: ${{ matrix.debian_revision }} localversion: ${{ matrix.localversion || '' }} kver-extra: ${{ matrix.kver_extra || '' }} + debug-build: ${{ inputs.debug-build || false }} + qcom-next-pr: ${{ inputs.qcom-next-pr || '' }} + kernel-topics-pr: ${{ inputs.kernel-topics-pr || '' }} diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index a96f4caa..b60b2d09 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -7,6 +7,10 @@ name: release # packages are promoted from the ephemeral Debusine CI workspace to the stable # target workspace (e.g. qli) via debusine-action lib/release. # +# A dispatch says only which entries to release: "all", or a comma-separated +# list of build names. Everything else about a release comes from the entry, +# so there is nothing here to type that the matrix has not already reviewed. +# # To update a release ref: open a PR that changes branch_or_tag on every # Release entry for that flavour in ci/build-matrix.yaml. A flavour's Release # entries must agree on the ref, so resolve-matrix.py rejects the matrix if @@ -18,23 +22,10 @@ name: release on: workflow_dispatch: inputs: - release-scope: - description: 'Choose every suite of one flavour, or named build(s)' - type: choice - required: true - default: 'Selected flavour (all suites)' - options: - - 'Selected flavour (all suites)' - - 'Selected builds' - flavour: - description: 'Kernel flavour to release in every suite it targets (Selected flavour)' - required: true - default: 'qcom-next' - type: string builds: - description: 'Comma-separated build names, e.g. qcom-next-trixie,qcom-next-forky (Selected builds)' + description: 'Builds to release: all, or comma-separated Release build names from ci/build-matrix.yaml (e.g. qcom-next-trixie,qcom-next-forky)' required: true - default: 'qcom-next-trixie' + default: 'all' type: string permissions: @@ -60,28 +51,21 @@ jobs: - name: Generate release matrix id: set-matrix env: - RELEASE_SCOPE_INPUT: ${{ github.event.inputs.release-scope }} - INPUT_FLAVOUR: ${{ github.event.inputs.flavour }} - INPUT_BUILDS: ${{ github.event.inputs.builds }} + INPUT_BUILDS: ${{ inputs.builds }} run: | set -euo pipefail - RELEASE_SCOPE="$RELEASE_SCOPE_INPUT" + BUILDS="${INPUT_BUILDS:-all}" - case "$RELEASE_SCOPE" in - "Selected flavour (all suites)") - echo "Source: ci/build-matrix.yaml (flavour: $INPUT_FLAVOUR, all suites)" - args=(--type Release --flavour "$INPUT_FLAVOUR") - ;; - "Selected builds") - echo "Source: ci/build-matrix.yaml (builds: $INPUT_BUILDS)" - args=(--type Release --build "$INPUT_BUILDS") - ;; - *) - echo "ERROR: unsupported release scope: $RELEASE_SCOPE" >&2 - exit 1 - ;; - esac + args=(--type Release) + if [[ "$BUILDS" == "all" ]]; then + echo "Source: ci/build-matrix.yaml (every Release entry)" + else + echo "Source: ci/build-matrix.yaml (builds: $BUILDS)" + # A name matching no Release entry fails here, so a typo stops the + # run rather than quietly releasing a smaller set than intended. + args+=(--build "$BUILDS") + fi MATRIX=$(ci/scripts/resolve-matrix.py "${args[@]}") echo "matrix=$MATRIX" >> "$GITHUB_OUTPUT" diff --git a/README.md b/README.md index 3fefeeb2..00117e0a 100644 --- a/README.md +++ b/README.md @@ -127,10 +127,10 @@ variant is a matrix change, not a workflow redesign. | Workflow | Purpose | Trigger | | --- | --- | --- | -| `daily.yml` | Resolves and runs the Daily matrix. | Scheduled daily at `23:00 UTC`, or manual dispatch. | +| `daily.yml` | Resolves and runs the Daily matrix. The manual build entry point. | Scheduled daily at `23:00 UTC`, or manual dispatch. | | `release.yml` | Resolves and runs the Release matrix. | Manual dispatch only. | -| `build-kernel-debian.yml` | Builds one Debian-suite entry in Debusine and publishes it to S3. | Manual dispatch or called by Daily and PR build. | -| `build-kernel-ubuntu.yml` | Builds one Ubuntu-suite entry on the Docker path and publishes it to S3. | Manual dispatch or called by Daily and PR build. | +| `build-kernel-debian.yml` | Builds one Debian-suite entry in Debusine and publishes it to S3. | Called by Daily and PR build. | +| `build-kernel-ubuntu.yml` | Builds one Ubuntu-suite entry on the Docker path and publishes it to S3. | Called by Daily and PR build. | | `release-kernel-debian.yml` | Builds one Debian-suite entry in Debusine and promotes it to the release workspace. | Called by Release. | The three build workflows share their steps through two composite actions @@ -153,12 +153,12 @@ not take. Daily is the recurring build and artifact-publication path. - The scheduled run resolves the full `Daily` matrix. -- A manual run selects one **Build scope**: - - **Full matrix** builds every configured variant and suite. - - **Selected builds** builds a comma-separated list of build names, e.g. - `qcom-next-trixie,qcom-next-debug-forky`. - - **Selected flavour (all suites)** builds one flavour in every suite it - targets. +- A manual run says which entries to build in one **Builds** field: + - `all`, the default, builds every configured variant and suite. + - A comma-separated list of build names builds those entries, e.g. + `qcom-next-trixie,qcom-next-debug-forky`. A name matching no `Daily` entry + fails the run rather than narrowing it, and both families are selected from + one list, so a mixed list starts Debian and Ubuntu legs from one dispatch. - `latest_tag` resolves the newest matching dated tag; `branch_tip` resolves the configured branch directly. - Debian suites build in Debusine, then their `.deb` outputs are downloaded and @@ -170,11 +170,11 @@ Daily is the recurring build and artifact-publication path. Release is the controlled promotion path. -- It is manual only and uses one **Release scope** for a kernel variant: - - **Selected flavour (all suites)** is the normal release action and promotes - every configured Release suite for that flavour. - - **Selected builds** promotes a comma-separated list of build names when a - targeted action is required. +- It is manual only, and says which entries to release in one **Builds** field, + the same way Daily does: + - `all`, the default, promotes every configured `Release` entry. + - A comma-separated list of build names promotes those entries when a + targeted action is required, e.g. `qcom-next-trixie,qcom-next-forky`. - It uses the pinned `branch_or_tag` from the selected `Release` matrix row; it never resolves a newest tag. - Debian source and binary artifacts are built in per-flavour, per-suite @@ -186,11 +186,10 @@ Release is the controlled promotion path. the release credential and enforces the required approval gate before promotion to `qli`. -Direct `build-kernel-debian.yml` and `build-kernel-ubuntu.yml` dispatches are -build-only: neither has a promotion path to offer. Release promotion is -initiated exclusively by `release.yml`, which owns the target workspace and -production release controls, and is the only caller of -`release-kernel-debian.yml`. +A `daily.yml` dispatch is build-only: it publishes to the daily S3 path and has +no promotion path to offer. Release promotion is initiated exclusively by +`release.yml`, which owns the target workspace and production release controls, +and is the only caller of `release-kernel-debian.yml`. Only the Debian family has a release path at all, because promotion runs through Debusine. `resolve-matrix.py` rejects a `Release` entry for any other @@ -206,7 +205,7 @@ them to the workflow matrix as they stand. Each entry carries: | Field | Purpose | | --- | --- | -| `name` | The name of this one build, and nothing else: its Actions job name, and what a manual dispatch asks for. Unique within a delivery type, so a Daily and its Release may share a name. Never reaches a package name, a version, or a published path. Lowercase letters, digits, and internal hyphens only. | +| `name` | The name of this one build, and nothing else: its Actions job name, and what a manual dispatch asks for. Unique within a delivery type, so a Daily and its Release may share a name. Never reaches a package name, a version, or a published path. Lowercase letters, digits, and internal hyphens only, and not `all`, which a dispatch reads as every entry. | | `type` | `Daily` or `Release`. | | `suite` | The one suite this entry builds for. | | `flavour` | The kernel's own identity, and the only matrix field that reaches the built kernel. `derive-localversion.sh` makes it the LOCALVERSION suffix, so two flavours built from one ref get distinct kernel releases and their `linux-image` packages coexist. All entries sharing a flavour build the same package for different suites. | @@ -257,11 +256,11 @@ face value: - Where a suite has both, its `Daily` revision is its `Release` revision plus a trailing `~`. -A direct dispatch of a build workflow has no matrix context of its own, so when -its `debian-revision` input is empty the `prepare-kernel-source` action looks up -the `Daily` entry for the build name and suite it was given (`resolve-matrix.py ---field debian_revision`) and builds at the revision the daily build would have -used. +The build workflows pass their entry's own `debian_revision` through. A caller +with no entry in hand can leave the `debian-revision` input empty, and the +`prepare-kernel-source` action looks up the `Daily` entry for the build name and +suite it was given (`resolve-matrix.py --field debian_revision`) and builds at +the revision the daily build would have used. Each entry has a distinct prepared-source artifact, Debusine child workspace, and S3 path keyed by `flavour + suite`. This prevents two flavours that both @@ -326,10 +325,9 @@ flowchart LR flowchart TD subgraph triggers[Triggers] A1["daily.yml\nScheduled full matrix"] - A2["daily.yml\nManual full or filtered builds"] + A2["daily.yml\nManual: all or named builds"] A5["pr-build.yml\nFull Daily matrix on every PR"] - A3["release.yml\nManual full or filtered builds"] - A4["build-kernel-debian.yml\nbuild-kernel-ubuntu.yml\nManual one-off build"] + A3["release.yml\nManual: all or named builds"] end subgraph matrix[Matrix entry points] @@ -362,7 +360,6 @@ flowchart TD B2 --> C2 B5 --> C2 B4 --> C2 - A4 --> C2 C2 --> C3 & C4 C3 --> C5 & C6 C4 --> D1 @@ -492,42 +489,40 @@ out-of-tree module builds are required. ## Manual Builds -Use **Actions** → **build-kernel-debian** or **build-kernel-ubuntu** for a -one-off build, picking the one that builds the suite you want. These are -explicit override workflows, not matrix-derived delivery flows: use `daily.yml` -and `release.yml` for normal Daily and Release operations. Neither promotes. +Use **Actions** → **daily** → **Run workflow** for a one-off build, and +**Actions** → **release** to promote. Both are dispatched the same way: one +**Builds** field naming what to run. -`build`, `suite`, and `ref-strategy` are the required build selection. -All remaining package, configuration, and PR inputs are advanced overrides for -validation or debugging. Build name and suite are free-text matrix values -rather than static dropdowns, so adding a matrix entry never requires editing -the workflow UI. A suite belonging to the other family is rejected by the -`prepare` job before anything is cloned or built, so the only cost of picking -the wrong workflow is a fast failure. - -The available inputs are: +| Input | Default | Purpose | +| --- | --- | --- | +| `builds` | `all` | `all` runs every entry of that workflow's delivery type. Otherwise a comma-separated list of build `name` values from `ci/build-matrix.yaml`, e.g. `qcom-next-trixie,qcom-next-debug-forky`. | + +Everything else about a build — its suite, flavour, kernel repository and ref, +package names, config fragments, DKMS modules and Debian revision — comes from +the entry, so there is nothing to retype and nothing to get wrong. Build names +are free-text matrix values rather than a static dropdown, so adding a matrix +entry never requires editing the workflow UI, and a name that matches no entry +of the delivery type fails the run with the list of names that do. `daily` +routes each selected entry to the workflow that builds its family, so one +dispatch can name Debian and Ubuntu builds together. + +`daily` carries three further inputs, which the matrix deliberately says +nothing about because they belong to a one-off validation run rather than to a +delivery target. They apply to every selected build, and a scheduled run leaves +them at their defaults: | Input | Default | Purpose | | --- | --- | --- | -| `build` | `qcom-next-trixie` | Names this build; also selects the matrix entry a blank `debian-revision` is taken from. | -| `flavour` | `qcom-next` | Kernel flavour: the LOCALVERSION suffix, and so the kernel release identity. | -| `suite` | `trixie` | Target suite. | -| `ref-strategy` | `latest_tag` | `latest_tag`, `branch_tip`, or `pinned_ref`. | -| `kernel-branch` | `qcom-next` | Branch for `branch_tip`, or immutable ref for `pinned_ref`; ignored by `latest_tag`. | -| `tag-pattern` | `qcom-next-*` | Tag glob for `latest_tag`; ignored by `branch_tip` and `pinned_ref`. | -| `kernel-url` | `qualcomm-linux/kernel` | Advanced alternate kernel repository. | -| `srcpkg` | `linux-qcom-next` | Advanced source package identity override. | -| `binpkg` | `linux-image-qcom-next` | Advanced image metapackage identity override. | -| `kernel-config` | Empty | Advanced extra fragments applied on top of all of `debian/config-available/`, e.g. `intree:arch/arm64/configs/qcom_debug.config`. | -| `dkms` | Empty | Advanced comma-separated DKMS modules bundled into the image package, each without the `-dkms` suffix. Empty bundles none. | -| `debian-revision` | The matrix Daily revision | Advanced Debian revision override. Left empty, the build takes the `debian_revision` of the matrix's `Daily` entry for the selected variant and suite; direct builds always use the Daily entry since they are build-only and non-promoting. | -| `localversion` | Auto-derived | Advanced explicit `LOCALVERSION` override. | -| `kver-extra` | Empty | Advanced kernel-release suffix. | -| `debug-build` | `false` | Advanced debug configuration toggle. | - -The workflow also supports advanced Qualcomm-only PR overrides for validation -builds. Direct builds are artifact builds; Release promotion is performed only -through `release.yml`. +| `debug-build` | `false` | Advanced debug configuration toggle. For a lasting debug kernel, use the `qcom-next-debug` flavour instead. | +| `qcom-next-pr` | Empty | Advanced Qualcomm-only override: `qcom-next` PR numbers to merge before building. | +| `kernel-topics-pr` | Empty | Advanced Qualcomm-only override: `kernel-topics` PR numbers to apply as patches. | + +A `daily` dispatch is an artifact build and publishes to the daily S3 path; +Release promotion is performed only through `release.yml`. The build workflows +themselves (`build-kernel-debian.yml`, `build-kernel-ubuntu.yml`, +`release-kernel-debian.yml`) are `workflow_call` only and cannot be dispatched: +one run of each is one matrix entry, and a reusable workflow cannot fan itself +out over a list. ## Configuration @@ -586,8 +581,9 @@ To add a new suite (for an existing or new variant): [Overview](#overview)), and so it does not collide with another entry building the same `srcpkg`. -No workflow dispatch choices need to be updated: manual Daily and Release -inputs accept matrix-defined variant and suite strings. +No workflow dispatch choices need to be updated: the Daily and Release +dispatches take build names as free text, so a new entry is dispatchable by +name, and is picked up by `all`, as soon as it is merged. Run `ci/scripts/resolve-matrix.py --type Daily` and `--type Release` locally to validate a matrix change before pushing it; both validate the whole document, diff --git a/ci/build-matrix.yaml b/ci/build-matrix.yaml index c668faeb..00d72d61 100644 --- a/ci/build-matrix.yaml +++ b/ci/build-matrix.yaml @@ -24,7 +24,8 @@ # A build's name is what CI calls it: the Actions job name, and what a manual # dispatch asks for by name. It never reaches a package name or version, and # never a published path. It must be unique within a delivery type, so a Daily -# and the Release that supersedes it share one. +# and the Release that supersedes it share one. It may not be "all", which the +# daily and release dispatch forms read as every entry of their type. # # dkms varies by suite because it depends on which -dkms packages the # target archive carries: the Debian suites bundle camx and iris-vpu, which diff --git a/ci/scripts/resolve-matrix.py b/ci/scripts/resolve-matrix.py index edadb6a9..7afa7d3f 100755 --- a/ci/scripts/resolve-matrix.py +++ b/ci/scripts/resolve-matrix.py @@ -159,6 +159,12 @@ def family_for(suite): NAME_RE = re.compile(r"^[a-z0-9]+(?:[a-z0-9-]*[a-z0-9])?$") +# The dispatch forms of daily.yml and release.yml take one builds field, where +# "all" means every entry of the delivery type and anything else is a list of +# names. A build actually called all would be unreachable through them, so the +# matrix may not define one. +RESERVED_NAMES = ("all",) + # A Debian revision: no hyphen (that would start a new revision component) and # none of the characters dpkg rejects in a version. REVISION_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9.+~]*$") @@ -270,6 +276,12 @@ def check_entry(entry, report): if isinstance(value, str) and not NAME_RE.match(value): report(f"{field} must use lowercase letters, digits, and internal hyphens") + if entry.get("name") in RESERVED_NAMES: + report( + f"name {entry['name']} is reserved by the build workflows' dispatch " + "form, where it selects every entry rather than one of them" + ) + check_kernel_config(entry, report) check_dkms(entry, report) From 5a5cbcc125a8a7e6c7d1749ce3ce8c2b96e2ea96 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 19/59] ci: build what a release ships, and promote it inside the build A release was a second build. release.yml took a pinned tag, built it from scratch in a fresh CI workspace, and promoted that -- bits the daily build had already produced from the same tag and tested. Everything else followed from the duplicate build: a second set of matrix entries to describe it, a type field to tell the two sets apart, ref_strategy rules per type, and a revision convention whose only job was to keep the two builds' versions apart. What shipped was never quite the artifact that passed the daily. Promoting a previous run's build instead is not open to us, and that is worth saying plainly because it is the constraint the whole design turns on: lib/build names the CI workspace after the run that created it (-gh----) and it does not outlive that run. By the time anyone decides to release, there is no longer a workspace to promote out of. Publishing is part of the build or it does not happen. So promotion moves into the build. build-kernel-debian.yml gains a promote job that publishes the run's own artifacts into the workspace its caller named, and the caller decides which archive that is and what stands in front of it: promote-environment defaults to Staging, where the nightly promotes unattended, and release.yml passes Production so an approval gate stands in front of qli. A PR build names no workspace and so reaches no archive. release.yml then builds the ref it releases, in the same workflow the nightly calls, and promotes that build while the run still holds its workspace. What reaches qli is the artifact this run produced and tested, from a ref that cannot have moved since it was reviewed. What is released is written down rather than typed in. ci/build-matrix.yaml gains a releases list beside builds: same shape, plus a target_workspace, and ref_strategy restricted to pinned_ref so that two dispatches of one entry cannot ship different kernels. Updating a release is a pull request that changes branch_or_tag, which puts the ref that ships under review before the run that ships it -- where a dispatch field put it at the mercy of whoever typed it. The two lists are validated together however few of them a caller selects, so a broken release entry fails a nightly run rather than lying in wait for whoever next tries to release. Releases are Debian-only, enforced rather than assumed: promotion runs through Debusine and the Ubuntu path has no workspace to promote into, so an Ubuntu entry there would name a destination nothing could deliver it to. That collapses builds to one kind of entry, 15 to 11. type is gone, and with it the per-type ref and revision rules. release-kernel-debian.yml goes too: it existed to be the separate release build, and there is no longer one. Signed-off-by: Christopher Obbard --- .github/actions/debusine-build/action.yml | 16 +- .../actions/prepare-kernel-source/action.yml | 42 +- .github/workflows/build-kernel-debian.yml | 215 +++++++- .github/workflows/daily.yml | 50 +- .github/workflows/pr-build.yml | 14 +- .github/workflows/release-kernel-debian.yml | 223 -------- .github/workflows/release.yml | 119 +++-- README.md | 497 ++++++++++-------- ci/build-matrix.yaml | 252 +++++---- ci/scripts/resolve-matrix.py | 336 ++++++------ 10 files changed, 975 insertions(+), 789 deletions(-) delete mode 100644 .github/workflows/release-kernel-debian.yml diff --git a/.github/actions/debusine-build/action.yml b/.github/actions/debusine-build/action.yml index e089dff9..17b3140a 100644 --- a/.github/actions/debusine-build/action.yml +++ b/.github/actions/debusine-build/action.yml @@ -3,11 +3,11 @@ description: > Generate the Debian source package from the prepared tree and submit it to Debusine, returning the CI workspace it was built in. -# Shared by the two Debian workflows, which differ only in what they do with -# the result: build-kernel-debian.yml downloads the .deb files and uploads them -# to S3, release-kernel-debian.yml promotes them to a target workspace. Holding -# these steps in a composite action lets each workflow declare its own runner, -# container and environment while keeping one copy of the build itself. +# Called by build-kernel-debian.yml, which decides what becomes of the result: +# every build is downloaded and uploaded to S3, and one whose entry names a +# target workspace is also promoted into it. Holding these steps in a composite +# action keeps the submission itself in one place, separate from the job that +# declares the runner, container and environment it happens in. # # The caller must have checked out pkg-linux-qcom into the workspace first, # which is also where this action comes from. Runs inside the @@ -41,9 +41,9 @@ inputs: extra-build-dep-workspaces: description: > Workspaces the build resolves its build-dependencies from, space - separated. A staging build may depend on a package version that has only - reached qli-staging; a release build reads qli alone, so a released - kernel is never built against something unreleased. + separated. The nightly build reads qli-staging as well as qli, because a + kernel may need a -dkms package version that has reached staging and not + yet been released. default: 'qli' debusine-action-ref: description: 'Ref of qualcomm-linux/debusine-action to use' diff --git a/.github/actions/prepare-kernel-source/action.yml b/.github/actions/prepare-kernel-source/action.yml index 9a993ad2..3811def2 100644 --- a/.github/actions/prepare-kernel-source/action.yml +++ b/.github/actions/prepare-kernel-source/action.yml @@ -20,6 +20,15 @@ description: > # kernel ref comes from a remote's tag list, and on pr-build.yml the inputs # come from the build matrix as the pull request wrote it. +outputs: + srcpkg_version: + description: > + Version of the source package this build will produce, read from the + generated debian/changelog. A caller that promotes the result into an + archive needs it before the build starts, to see whether that version is + already published there. + value: ${{ steps.version.outputs.srcpkg_version }} + inputs: build: description: 'Name of this build, as listed in ci/build-matrix.yaml' @@ -294,10 +303,9 @@ runs: - name: Resolve Debian revision # The build workflows pass the matrix entry's own debian-revision # straight through, so this lookup is the fallback for a caller that has - # no entry in hand and leaves the input empty: it looks up the Daily - # entry for the build name and suite it was given, and builds at the - # revision the daily build would have used. Always the Daily entry -- - # a release is driven by release.yml, which passes its own revision. + # no entry in hand and leaves the input empty: it looks up the entry for + # the build name and suite it was given, and builds at the revision the + # nightly build uses. id: revision shell: bash env: @@ -311,11 +319,10 @@ runs: echo "Using supplied Debian revision: $DEBIAN_REVISION" else DEBIAN_REVISION=$(ci/scripts/resolve-matrix.py \ - --type Daily \ --build "$BUILD_INPUT" \ --suite "$DISTRO" \ --field debian_revision) || { - echo "::error::No Daily matrix entry named $BUILD_INPUT for suite $DISTRO." + echo "::error::No matrix entry named $BUILD_INPUT for suite $DISTRO." echo "::error::Add one to ci/build-matrix.yaml, or set the debian-revision input to build this combination ad hoc." exit 1 } @@ -426,6 +433,29 @@ runs: "ghcr.io/qualcomm-linux/pkg-builder:$DOCKER_DISTRO" \ ./prepare-source.sh "${ARGS[@]}" + - name: Read the generated source version + # prepare-source.sh has just written debian/changelog, so the version + # this build will produce is now decided and can be reported to the + # caller. Taken from the changelog rather than recomputed here: that file + # is what the source package is built from, so nothing can disagree with + # it. Parsed with sed rather than dpkg-parsechangelog because these steps + # run on the runner host, which is not required to have dpkg-dev; the + # first line of a changelog is "srcpkg (version) suite; urgency=...". + id: version + shell: bash + env: + KERNEL_DIR: ${{ steps.clone.outputs.kernel_dir }} + run: | + set -euo pipefail + SRCPKG_VERSION=$(sed -n '1s/^[^ ]* (\([^)]*\)).*/\1/p' \ + "$KERNEL_DIR/debian/changelog") + [[ -n "$SRCPKG_VERSION" ]] || { + echo "::error::Could not read a version from the generated debian/changelog." + exit 1 + } + echo "Source version: $SRCPKG_VERSION" + echo "srcpkg_version=$SRCPKG_VERSION" >> "$GITHUB_OUTPUT" + - name: Pack prepared source tree shell: bash env: diff --git a/.github/workflows/build-kernel-debian.yml b/.github/workflows/build-kernel-debian.yml index e3e2ca1a..46ff68da 100644 --- a/.github/workflows/build-kernel-debian.yml +++ b/.github/workflows/build-kernel-debian.yml @@ -1,12 +1,13 @@ name: build-kernel-debian -# Build a kernel package for a Debian-family suite and publish it to S3. +# Build a kernel package for a Debian-family suite, publish it to S3, and +# promote it into a Debusine workspace when the caller names one. # # Called per matrix entry by daily.yml and pr-build.yml, which select the # Debian entries with resolve-matrix.py --family debian and call only this # workflow for them. Nothing here is conditional on the suite: a caller that # reaches this file has already decided the build belongs on the Debusine -# path, so every job in it runs. +# path. # # workflow_call only. One run of this workflow is one matrix entry, and a # reusable workflow cannot fan itself out over several, so manual builds are @@ -14,13 +15,19 @@ name: build-kernel-debian # per Debian entry it selected. That also keeps every value describing a build # in ci/build-matrix.yaml, rather than in a dispatch form that drifts from it. # -# release-kernel-debian.yml is the same build with the release tail: it -# promotes to a target Debusine workspace instead of publishing to S3. +# Publishing happens here and nowhere else. A caller that names a +# target-workspace has this run's own build promoted into it, out of the +# ephemeral CI workspace it was built in, so the archive carries the artifact +# this run produced and tested. release.yml is such a caller: it builds the ref +# it releases rather than promoting a build some earlier run made. # # Jobs: -# prepare (self-hosted): clone, patch, inject debian/, generate the source. +# prepare (self-hosted): clone, patch, inject debian/, generate the source, +# and decide whether the version it produces is already published. # build (debusine-pkg-builder container): submit to Debusine. # publish (self-hosted): download the .deb files and upload them to S3. +# promote (debusine-pkg-builder container): copy the result into +# target-workspace. Skipped entirely when no workspace was named. on: workflow_call: @@ -89,6 +96,14 @@ on: description: 'Parent Debusine workspace for CI child workspace creation' type: string default: '' + target-workspace: + description: > + Debusine workspace to promote the built package into, empty to + promote nowhere. Set from the matrix entry's target_workspace by the + nightly build; left empty by pr-build.yml, whose builds must not + reach an archive. + type: string + default: '' qcom-next-pr: description: 'qcom-next PR numbers to merge (e.g. "42 43")' type: string @@ -97,6 +112,15 @@ on: description: 'kernel-topics PR numbers to apply as patches' type: string default: '' + promote-environment: + description: > + GitHub environment the promote job runs in, and so which approvals + and which credentials stand between a build and the archive it + publishes into. Defaults to Staging, where the nightly promotes + unattended. release.yml passes Production, because a release reaches + an archive people install from and an unreviewed one must not. + type: string + default: 'Staging' secrets: DEBUSINE_USER: description: 'Debusine account used to submit CI builds' @@ -111,7 +135,15 @@ permissions: jobs: # --------------------------------------------------------------------------- - # prepare: clone kernel source, inject packaging, run prepare-source.sh. + # prepare: clone kernel source, inject packaging, run prepare-source.sh, and + # report the version that produces. + # + # The version is decided here, by the ref this run resolved and the revision + # it was given, so this is the first point at which the run can tell whether + # there is anything new to build. When the caller named a target workspace + # and that version is already in it, every later job is skipped: the nightly + # build of a tag that has not moved would otherwise rebuild the same version + # and then fail promoting it as a duplicate. # --------------------------------------------------------------------------- prepare: name: Generate source package @@ -119,6 +151,9 @@ jobs: permissions: contents: read # actions/checkout packages: read # docker pull ghcr.io/qualcomm-linux/pkg-builder + outputs: + srcpkg_version: ${{ steps.prepare.outputs.srcpkg_version }} + already_published: ${{ steps.published.outputs.already_published }} steps: # debian/ and ci/ live on the same branch, so one checkout supplies both # the packaging and the scripts, the delivery matrix they read, and the @@ -132,6 +167,7 @@ jobs: persist-credentials: false - name: Prepare kernel source + id: prepare uses: ./.github/actions/prepare-kernel-source with: build: ${{ inputs.build }} @@ -153,12 +189,91 @@ jobs: qcom-next-pr: ${{ inputs.qcom-next-pr }} kernel-topics-pr: ${{ inputs.kernel-topics-pr }} + # Only needed to ask the target workspace what it already has, so it is + # checked out only when there is a target workspace to ask. + - name: Checkout debusine-action helpers + if: ${{ inputs.target-workspace != '' }} + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + repository: qualcomm-linux/debusine-action + ref: main + path: debusine-action + fetch-depth: 1 + sparse-checkout: | + lib + + # The archive is the authority on whether there is anything to do. A + # rerun, a retry, and a night on which the tracked tag did not move all + # generate the version that is already published, because the version is + # a function of the resolved ref and the revision -- so asking here is + # what turns "nothing new upstream" into a skipped run rather than a + # rebuild that fails promoting a duplicate. + # + # Two ways of not getting an answer, deliberately treated differently. + # A workspace whose signing key cannot be fetched is a configuration + # problem -- it does not exist, or this token cannot see it -- and fails + # the run, because every later night would fail the same way and silence + # would hide it. An index that cannot be read once the workspace is + # reachable is reported as not having the version, which builds and lets + # the promotion fail loudly. Neither is allowed to look like a + # successful night on which nothing needed doing. + - name: Check the target workspace for this version + id: published + if: ${{ inputs.target-workspace != '' }} + env: + DEBUSINE_HOST: ${{ vars.DEBUSINE_HOST }} + DEBUSINE_SCOPE: ${{ vars.DEBUSINE_SCOPE }} + DEBUSINE_USER: ${{ secrets.DEBUSINE_USER }} + DEBUSINE_TOKEN: ${{ secrets.DEBUSINE_TOKEN }} + DEBUSINE_WORKSPACE: ${{ inputs.target-workspace }} + SUITE: ${{ inputs.suite }} + SRCPKG: ${{ inputs.srcpkg }} + SRCPKG_VERSION: ${{ steps.prepare.outputs.srcpkg_version }} + run: | + # No -x: DEBUSINE_TOKEN is in env, keep xtrace off. + set -euo pipefail + debusine-action/lib/generate-apt-config || { + echo "::error::Cannot read $DEBUSINE_WORKSPACE. Check that the workspace exists and that DEBUSINE_TOKEN may read it." + exit 1 + } + sudo apt-get install -y --no-install-recommends devscripts + chdist create target-workspace + rm -f ~/.chdist/target-workspace/etc/apt/sources.list + install -d ~/.chdist/target-workspace/etc/apt/sources.list.d + install -d ~/.chdist/target-workspace/etc/apt/auth.conf.d + install -m 0644 debusine-ci.sources \ + ~/.chdist/target-workspace/etc/apt/sources.list.d/ + install -m 0600 debusine-ci-auth.conf \ + ~/.chdist/target-workspace/etc/apt/auth.conf.d/ + + if ! chdist apt-get target-workspace update; then + echo "::warning::Could not read $DEBUSINE_WORKSPACE; treating $SRCPKG $SRCPKG_VERSION as unpublished." + echo "already_published=false" >> "$GITHUB_OUTPUT" + exit 0 + fi + + if chdist apt-cache target-workspace showsrc "$SRCPKG" 2>/dev/null \ + | grep -qxF "Version: $SRCPKG_VERSION"; then + echo "$SRCPKG $SRCPKG_VERSION is already in $DEBUSINE_WORKSPACE." + echo "Nothing new to build; the rest of this run is skipped." \ + >> "$GITHUB_STEP_SUMMARY" + echo "already_published=true" >> "$GITHUB_OUTPUT" + else + echo "$SRCPKG $SRCPKG_VERSION is not in $DEBUSINE_WORKSPACE yet." + echo "already_published=false" >> "$GITHUB_OUTPUT" + fi + # --------------------------------------------------------------------------- # build: generate the Debian source package and submit it to Debusine. + # + # Skipped, with everything after it, when prepare found this version already + # published in the target workspace. # --------------------------------------------------------------------------- build: name: Build package (Debusine) needs: prepare + if: ${{ needs.prepare.outputs.already_published != 'true' }} runs-on: ubuntu-latest permissions: contents: read # actions/checkout @@ -172,6 +287,7 @@ jobs: password: ${{ secrets.GITHUB_TOKEN }} outputs: workspace: ${{ steps.debusine.outputs.workspace }} + srcpkg_version: ${{ steps.debusine.outputs.srcpkg_version }} steps: # Only for the debusine-build action itself: the package being built # comes from the prepared-source artifact, not from this checkout. @@ -261,3 +377,90 @@ jobs: s3_bucket: ${{ vars.ARTIFACT_S3_BUCKET }} path: deb-artifacts destination: ${{ github.repository_owner }}/pkg/debusine/${{ github.event.repository.name }}/${{ inputs.flavour }}/${{ inputs.suite }}/${{ github.run_id }}-${{ github.run_attempt }}/ + + # --------------------------------------------------------------------------- + # promote: copy the built package from the ephemeral CI workspace into the + # workspace the caller named, making it installable from that archive. + # + # This is what replaces a separate release build. The artifact promoted here + # is the one the build above produced, so an archive never carries a version + # that was not built and published by this run. + # + # Promotion is in-run by necessity, not by preference: lib/build names the + # CI workspace after the run that created it + # (-gh----) and it does not outlive it, so + # the only moment its contents can be published is while the run still holds + # it. That is why release.yml builds the ref it releases rather than + # promoting a nightly afterwards -- by then there would be nothing left to + # promote from. + # + # The if: is the whole of the decision. A caller that named no workspace + # promotes nowhere, and a run carrying any of the validation overrides + # promotes nowhere either: those builds are deliberately not the kernel the + # matrix describes, and an archive must only ever receive that one. Guarding + # it here rather than in the caller means it holds for every caller, + # including a future one that forwards target-workspace without thinking + # about the overrides. + # + # Which archive, and what stands in front of it, is the caller's: + # promote-environment carries the approval gate, so a release into qli can + # require one where the nightly into staging does not. + # --------------------------------------------------------------------------- + promote: + name: Promote to ${{ inputs.target-workspace }} + # build alone: it already needs prepare, so a run prepare skipped as + # already published skips the build and this job with it. + needs: build + if: >- + ${{ inputs.target-workspace != '' + && !inputs.debug-build + && inputs.qcom-next-pr == '' + && inputs.kernel-topics-pr == '' }} + runs-on: ubuntu-latest + permissions: + contents: read # actions/checkout + packages: read # container: ghcr.io/qualcomm-linux/debusine-pkg-builder + environment: ${{ inputs.promote-environment }} + container: + image: ghcr.io/qualcomm-linux/debusine-pkg-builder:trixie + options: --user 0:0 + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + defaults: + run: + shell: bash + steps: + - name: Checkout debusine-action helpers + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + repository: qualcomm-linux/debusine-action + ref: main + path: debusine-action + fetch-depth: 1 + sparse-checkout: | + lib + + - name: Promote packages to the target workspace + env: + DEBUSINE_HOST: ${{ vars.DEBUSINE_HOST }} + DEBUSINE_SCOPE: ${{ vars.DEBUSINE_SCOPE }} + DEBUSINE_TOKEN: ${{ secrets.DEBUSINE_TOKEN }} + DEBUSINE_CI_WORKSPACE: ${{ needs.build.outputs.workspace }} + DEBUSINE_TARGET_WORKSPACE: ${{ inputs.target-workspace }} + SRCPKG_NAME: ${{ inputs.srcpkg }} + SRCPKG_VERSION: ${{ needs.build.outputs.srcpkg_version }} + SUITE: ${{ inputs.suite }} + run: | + set -eu + debusine-action/lib/release + + - name: Note the promoted version + env: + SRCPKG_NAME: ${{ inputs.srcpkg }} + SRCPKG_VERSION: ${{ needs.build.outputs.srcpkg_version }} + TARGET_WORKSPACE: ${{ inputs.target-workspace }} + run: | + echo "Promoted $SRCPKG_NAME $SRCPKG_VERSION to $TARGET_WORKSPACE" \ + >> "$GITHUB_STEP_SUMMARY" diff --git a/.github/workflows/daily.yml b/.github/workflows/daily.yml index 6c06dd8c..e9d06fc1 100644 --- a/.github/workflows/daily.yml +++ b/.github/workflows/daily.yml @@ -1,23 +1,30 @@ name: daily # Daily kernel package builds. Runs at 3:00 PM PST (23:00 UTC), one parallel -# build per Daily entry in ci/build-matrix.yaml. +# build per entry in ci/build-matrix.yaml. # # This is also the manual build entry point. Everything about a build comes # from its matrix entry, so a dispatch only says which entries to run: "all", # or a comma-separated list of build names. Build names are matrix values, so # a new one needs no workflow edit. # +# This is the only workflow that builds a kernel on a schedule: an entry +# naming a target_workspace is promoted into that workspace as part of its +# build, so what the staging archive carries is always something this workflow +# built and tested. Nothing promotes afterwards -- the CI workspace a build +# runs in is named after that run and does not outlive it. release.yml builds +# the ref it releases for the same reason. +# # Matrix schema: ci/build-matrix.yaml -# One entry is one generated package, so ci/scripts/resolve-matrix.py selects -# the entries with type: Daily and passes them through as they stand. Every -# field (name, suite, flavour, srcpkg, binpkg, kernel_config, dkms, -# debian_revision, git_clone, branch_or_tag, ref_strategy) becomes an input -# of the build workflow for the entry's family. name labels the job; flavour -# is what reaches the built kernel's version and its published paths. +# One entry is one generated package, so ci/scripts/resolve-matrix.py passes +# the selected entries through as they stand. Every field (name, suite, +# flavour, srcpkg, binpkg, kernel_config, dkms, debian_revision, git_clone, +# branch_or_tag, ref_strategy, target_workspace) becomes an input of the +# build workflow for the entry's family. name labels the job; flavour is what +# reaches the built kernel's version and its published paths. # -# To add a daily build target: add an entry to ci/build-matrix.yaml with -# type: Daily. No workflow changes needed. +# To add a daily build target: add an entry to ci/build-matrix.yaml. No +# workflow changes needed. on: schedule: @@ -26,7 +33,7 @@ on: workflow_dispatch: inputs: builds: - description: 'Builds to run: all, or comma-separated Daily build names from ci/build-matrix.yaml (e.g. qcom-next-trixie,qcom-next-debug-forky)' + description: 'Builds to run: all, or comma-separated build names from ci/build-matrix.yaml (e.g. qcom-next-trixie,qcom-next-debug-forky)' required: true default: 'all' type: string @@ -56,12 +63,11 @@ permissions: jobs: # --------------------------------------------------------------------------- - # configure-matrix: select the Daily entries from ci/build-matrix.yaml. + # configure-matrix: select the entries to build from ci/build-matrix.yaml. # # Delegates to ci/scripts/resolve-matrix.py, which validates the whole matrix - # and returns the entries with type: Daily. A dispatch narrows that to a - # comma-separated list of build names; "all", and a scheduled run, take the - # whole Daily matrix. + # and returns its entries. A dispatch narrows that to a comma-separated list + # of build names; "all", and a scheduled run, take the whole matrix. # --------------------------------------------------------------------------- configure-matrix: name: Configure build matrix @@ -84,16 +90,16 @@ jobs: set -euo pipefail # A scheduled run carries no inputs, so it lands on the dispatch - # default: the whole Daily matrix. + # default: the whole matrix. BUILDS="${INPUT_BUILDS:-all}" - args=(--type Daily) + args=() if [[ "$BUILDS" == "all" ]]; then - echo "Source: ci/build-matrix.yaml (every Daily entry)" + echo "Source: ci/build-matrix.yaml (every entry)" else echo "Source: ci/build-matrix.yaml (builds: $BUILDS)" - # A name matching no Daily entry fails here, so a typo stops the - # run rather than quietly narrowing it to nothing. + # A name matching no entry fails here, so a typo stops the run + # rather than quietly narrowing it to nothing. args+=(--build "$BUILDS") fi @@ -154,6 +160,12 @@ jobs: localversion: ${{ matrix.localversion || '' }} kver-extra: ${{ matrix.kver_extra || '' }} debusine-parent-workspace: ${{ matrix.debusine_parent_workspace || '' }} + # An entry naming a workspace is promoted into it once its build + # succeeds; one that names none is built and published to S3 only. The + # entry is forwarded as written: build-kernel-debian.yml is what declines + # to promote a run carrying the validation overrides below, so a build + # that is not the plain nightly cannot reach an archive by any caller. + target-workspace: ${{ matrix.target_workspace || '' }} # Dispatch-only validation overrides. A scheduled run carries no inputs, # so each falls back to the off value and builds the matrix as written. debug-build: ${{ inputs.debug-build || false }} diff --git a/.github/workflows/pr-build.yml b/.github/workflows/pr-build.yml index 8dfeea50..3ef53b0a 100644 --- a/.github/workflows/pr-build.yml +++ b/.github/workflows/pr-build.yml @@ -20,7 +20,7 @@ permissions: jobs: # --------------------------------------------------------------------------- - # configure-matrix: select the Daily entries from the PR's build-matrix.yaml. + # configure-matrix: select the entries from the PR's build-matrix.yaml. # # The PR head, not the merge commit: a push mid-run must not change what this # run tested. A SHA also resolves for fork PRs, whose head branch does not @@ -48,7 +48,7 @@ jobs: # so a PR gets one job per leg it actually builds and no skipped job # for the path that leg does not take. for family in debian ubuntu; do - MATRIX=$(ci/scripts/resolve-matrix.py --type Daily \ + MATRIX=$(ci/scripts/resolve-matrix.py \ --family "$family" --allow-empty) echo "$family=$MATRIX" >> "$GITHUB_OUTPUT" echo "$family entries: $(echo "$MATRIX" | jq length)" @@ -59,14 +59,16 @@ jobs: # daily.yml. # # A PR gets one check per build leg and nothing else: neither workflow has a - # branch to skip, and a PR builds the whole Daily matrix, so both families - # have entries and neither guard fires. + # branch to skip, and a PR builds the whole matrix, so both families have + # entries and neither guard fires. # # fail-fast is disabled so a broken trixie leg still leaves forky and # resolute results on the PR. # - # Neither takes DEBUSINE_RELEASE_TOKEN: a PR build never promotes, so it - # calls the S3-publishing workflow and not release-kernel-debian.yml. + # target-workspace is deliberately not passed on. An entry carrying one is + # promoted into it by the nightly build; a PR build of the same entry only + # has to prove it builds, and must not put a pull request's kernel into an + # archive anyone installs from. # --------------------------------------------------------------------------- build-debian: name: ${{ matrix.name }} diff --git a/.github/workflows/release-kernel-debian.yml b/.github/workflows/release-kernel-debian.yml deleted file mode 100644 index f4eeb40a..00000000 --- a/.github/workflows/release-kernel-debian.yml +++ /dev/null @@ -1,223 +0,0 @@ -name: release-kernel-debian - -# Build a kernel package for a Debian-family suite and promote it to a release -# workspace in Debusine. -# -# Called per matrix entry by release.yml. Identical to build-kernel-debian.yml -# up to the Debusine build -- both share the prepare-kernel-source and -# debusine-build actions -- and differs only in the tail: this one promotes the -# built package to target-workspace rather than publishing it to the daily S3 -# path. Which tail a run wants is decided by the workflow that calls it, so -# neither file carries the other's tail as a job to be skipped. -# -# Only the Debian family has a release path: promotion runs through Debusine, -# and resolve-matrix.py rejects a Release entry for any other suite. -# -# Jobs: -# prepare (self-hosted): clone, patch, inject debian/, generate the source. -# build (debusine-pkg-builder container): submit to Debusine. -# release (debusine-pkg-builder container): promote to the target workspace. - -on: - workflow_call: - inputs: - build: - description: 'Name of this build, as listed in ci/build-matrix.yaml' - type: string - default: 'qcom-next-trixie' - flavour: - description: 'Kernel flavour: the LOCALVERSION suffix and so the kernel release identity' - type: string - default: 'qcom-next' - suite: - description: 'Target Debian suite (trixie, forky, sid)' - type: string - default: 'trixie' - kernel-branch: - description: 'Immutable ref to build, from the entry branch_or_tag' - type: string - default: 'qcom-next' - git-clone: - description: 'Kernel repository URL' - type: string - default: 'https://github.com/qualcomm-linux/kernel' - ref-strategy: - description: 'Kernel ref strategy: a Release entry always pins one' - type: string - default: 'pinned_ref' - tag-pattern: - description: 'Tag glob, unused by pinned_ref and carried for symmetry with the Daily inputs' - type: string - default: '' - srcpkg: - description: 'Source package name' - type: string - default: 'linux-qcom-next' - binpkg: - description: 'Binary metapackage name' - type: string - default: 'linux-image-qcom-next' - kernel-config: - description: 'Extra config fragments applied on top of all of debian/config-available/' - type: string - default: '' - dkms: - description: 'Out-of-tree DKMS modules to bundle, comma-separated and without the -dkms suffix' - type: string - default: '' - debian-revision: - description: 'Debian revision component of the package version' - type: string - default: '0qli' - localversion: - description: 'Override LOCALVERSION suffix (auto-derived from the resolved ref if empty)' - type: string - default: '' - kver-extra: - description: 'Extra suffix appended to the package version' - type: string - default: '' - debusine-parent-workspace: - description: 'Parent Debusine workspace for CI child workspace creation' - type: string - default: '' - target-workspace: - description: 'Debusine workspace to promote the built package into' - type: string - required: true - secrets: - DEBUSINE_USER: - description: 'Debusine account used to submit release builds' - required: false - DEBUSINE_TOKEN: - description: 'Debusine token for the CI child workspace' - required: false - DEBUSINE_RELEASE_TOKEN: - description: 'Debusine token for promotion to the release target workspace' - required: false - -permissions: - contents: read - packages: read - -jobs: - # --------------------------------------------------------------------------- - # prepare: clone kernel source, inject packaging, run prepare-source.sh. - # --------------------------------------------------------------------------- - prepare: - name: Generate source package - runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] - permissions: - contents: read # actions/checkout - packages: read # docker pull ghcr.io/qualcomm-linux/pkg-builder - steps: - - name: Checkout pkg-linux-qcom - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - persist-credentials: false - - - name: Prepare kernel source - uses: ./.github/actions/prepare-kernel-source - with: - build: ${{ inputs.build }} - suite: ${{ inputs.suite }} - expect-family: debian - flavour: ${{ inputs.flavour }} - kernel-url: ${{ inputs.git-clone }} - ref-strategy: ${{ inputs.ref-strategy }} - kernel-branch: ${{ inputs.kernel-branch }} - tag-pattern: ${{ inputs.tag-pattern }} - srcpkg: ${{ inputs.srcpkg }} - binpkg: ${{ inputs.binpkg }} - kernel-config: ${{ inputs.kernel-config }} - dkms: ${{ inputs.dkms }} - debian-revision: ${{ inputs.debian-revision }} - localversion: ${{ inputs.localversion }} - kver-extra: ${{ inputs.kver-extra }} - - # --------------------------------------------------------------------------- - # build: generate the Debian source package and submit it to Debusine. - # --------------------------------------------------------------------------- - build: - name: Build package (Debusine) - needs: prepare - runs-on: ubuntu-latest - permissions: - contents: read # actions/checkout - packages: read # container: ghcr.io/qualcomm-linux/debusine-pkg-builder - environment: Production - container: - image: ghcr.io/qualcomm-linux/debusine-pkg-builder:trixie - options: --user 0:0 - credentials: - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - outputs: - workspace: ${{ steps.debusine.outputs.workspace }} - srcpkg_version: ${{ steps.debusine.outputs.srcpkg_version }} - steps: - - name: Checkout pkg-linux-qcom - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - persist-credentials: false - - - name: Build in Debusine - id: debusine - uses: ./.github/actions/debusine-build - with: - suite: ${{ inputs.suite }} - flavour: ${{ inputs.flavour }} - debusine-host: ${{ vars.DEBUSINE_HOST }} - debusine-scope: ${{ vars.DEBUSINE_SCOPE }} - debusine-user: ${{ secrets.DEBUSINE_USER }} - debusine-token: ${{ secrets.DEBUSINE_TOKEN }} - debusine-parent-workspace: ${{ inputs.debusine-parent-workspace || vars.DEBUSINE_PARENT_WORKSPACE || 'qli-ci' }} - # A release build reads qli alone, so a released kernel is never - # built against something that has not been released alongside it. - extra-build-dep-workspaces: qli - - # --------------------------------------------------------------------------- - # release: promote the built package from the CI workspace to the target. - # --------------------------------------------------------------------------- - release: - name: Release to Debusine - needs: build - runs-on: ubuntu-latest - permissions: - contents: read # actions/checkout - packages: read # container: ghcr.io/qualcomm-linux/debusine-pkg-builder - environment: Production - container: - image: ghcr.io/qualcomm-linux/debusine-pkg-builder:trixie - options: --user 0:0 - credentials: - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - defaults: - run: - shell: bash - steps: - - name: Checkout debusine-action helpers - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - persist-credentials: false - repository: qualcomm-linux/debusine-action - ref: main - path: debusine-action - fetch-depth: 1 - sparse-checkout: | - lib - - - name: Promote packages to target workspace - env: - DEBUSINE_HOST: ${{ vars.DEBUSINE_HOST }} - DEBUSINE_SCOPE: ${{ vars.DEBUSINE_SCOPE }} - DEBUSINE_TOKEN: ${{ secrets.DEBUSINE_RELEASE_TOKEN }} - DEBUSINE_CI_WORKSPACE: ${{ needs.build.outputs.workspace }} - DEBUSINE_TARGET_WORKSPACE: ${{ inputs.target-workspace }} - SRCPKG_NAME: ${{ inputs.srcpkg }} - SRCPKG_VERSION: ${{ needs.build.outputs.srcpkg_version }} - SUITE: ${{ inputs.suite }} - run: | - set -ex - debusine-action/lib/release diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index b60b2d09..0e04d5c9 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -1,29 +1,35 @@ name: release -# Release kernel package builds. Triggered manually only. +# Release kernel packages into the released archive. # -# Reads the Release entries from ci/build-matrix.yaml and builds each one using -# the matrix-selected ref strategy and branch_or_tag. After a successful build, -# packages are promoted from the ephemeral Debusine CI workspace to the stable -# target workspace (e.g. qli) via debusine-action lib/release. +# A release builds the ref it releases. It cannot promote a nightly instead: +# lib/build names the CI workspace after the run that created it and that +# workspace does not outlive the run, so by the time anyone decides to release, +# there is no longer anything to promote from. Building here means the artifact +# that reaches the archive is the one this run produced and tested, and the +# ref it came from is pinned rather than resolved, so two dispatches of one +# entry release the same kernel. # -# A dispatch says only which entries to release: "all", or a comma-separated -# list of build names. Everything else about a release comes from the entry, -# so there is nothing here to type that the matrix has not already reviewed. +# Every value describing a release lives in ci/build-matrix.yaml under +# "releases": the ref, the suites, the packaging, and the workspace each entry +# publishes into. A dispatch only says which of them to run. Updating a release +# is therefore a pull request that changes branch_or_tag, which is what puts +# the ref that ships under review -- rather than a version typed into this form +# at the moment of releasing. # -# To update a release ref: open a PR that changes branch_or_tag on every -# Release entry for that flavour in ci/build-matrix.yaml. A flavour's Release -# entries must agree on the ref, so resolve-matrix.py rejects the matrix if -# only some of them are updated. +# Debian suites only. Promotion runs through Debusine, and the Ubuntu path +# publishes to S3 with no workspace to promote into; resolve-matrix.py rejects +# a non-Debian release entry, so there is no Ubuntu leg to select here. # -# This workflow intentionally has no schedule trigger. Releases are always -# initiated manually after validation. +# Triggered manually only, and never on a schedule: a release is a decision. +# The promote job runs in the Production environment, so whatever approval that +# environment requires stands between this run and the archive. on: workflow_dispatch: inputs: builds: - description: 'Builds to release: all, or comma-separated Release build names from ci/build-matrix.yaml (e.g. qcom-next-trixie,qcom-next-forky)' + description: 'Releases to run: all, or comma-separated build names from ci/build-matrix.yaml (e.g. qcom-next-trixie,qcom-next-forky)' required: true default: 'all' type: string @@ -34,7 +40,14 @@ permissions: jobs: # --------------------------------------------------------------------------- - # configure-matrix: select the Release entries from ci/build-matrix.yaml. + # configure-matrix: select the entries to release from ci/build-matrix.yaml. + # + # --releases reads the releases list rather than builds. Both are validated + # whichever is selected, so a broken release entry fails here rather than + # part-way through a release. + # + # No --allow-empty and no family split: every release entry is Debian, so a + # selection that matches nothing is a mistake in the request and fails. # --------------------------------------------------------------------------- configure-matrix: name: Configure release matrix @@ -42,7 +55,7 @@ jobs: permissions: contents: read # actions/checkout outputs: - matrix: ${{ steps.set-matrix.outputs.matrix }} + releases: ${{ steps.set-matrix.outputs.releases }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: @@ -57,44 +70,61 @@ jobs: BUILDS="${INPUT_BUILDS:-all}" - args=(--type Release) + args=() if [[ "$BUILDS" == "all" ]]; then - echo "Source: ci/build-matrix.yaml (every Release entry)" + echo "Source: ci/build-matrix.yaml releases (every entry)" else - echo "Source: ci/build-matrix.yaml (builds: $BUILDS)" - # A name matching no Release entry fails here, so a typo stops the - # run rather than quietly releasing a smaller set than intended. + echo "Source: ci/build-matrix.yaml releases (builds: $BUILDS)" + # A name matching no release entry fails here, so releasing under a + # stale or mistyped name stops the run rather than quietly + # releasing a subset of what was asked for. args+=(--build "$BUILDS") fi - MATRIX=$(ci/scripts/resolve-matrix.py "${args[@]}") - echo "matrix=$MATRIX" >> "$GITHUB_OUTPUT" - echo "Matrix entries: $(echo "$MATRIX" | jq length)" + MATRIX=$(ci/scripts/resolve-matrix.py --releases "${args[@]}") + + echo "releases=$MATRIX" >> "$GITHUB_OUTPUT" + echo "Release entries: $(echo "$MATRIX" | jq length)" + + - name: Summarise what this run will release + env: + MATRIX: ${{ steps.set-matrix.outputs.releases }} + run: | + set -euo pipefail + # The refs and destinations in one place, so the approval on the + # Production environment is given against what is actually shipping + # rather than against a run number. + { + echo "### Releasing" + echo + echo "| Build | Suite | Ref | Revision | Into |" + echo "| --- | --- | --- | --- | --- |" + jq -r '.[] | "| \(.name) | \(.suite) | `\(.branch_or_tag)` | `\(.debian_revision)` | \(.target_workspace) |"' <<<"$MATRIX" + } >> "$GITHUB_STEP_SUMMARY" # --------------------------------------------------------------------------- - # build: one parallel job per Release matrix entry. + # release: one parallel job per selected entry. # - # Uses the ref strategy and branch_or_tag selected by the matrix. - # release-kernel-debian.yml builds exactly as the daily workflow does and - # then promotes to target_workspace, rather than publishing to the daily S3 - # path. Choosing the workflow here is what makes that the whole of the run: - # no job in it exists to publish the other way and be skipped. + # The same workflow the nightly calls, differing only in what it is told: a + # pinned ref instead of a resolved one, and the released workspace instead of + # staging. Nothing about how a kernel is built changes because it is being + # released. # - # There is no Ubuntu counterpart. Promotion runs through Debusine, which - # builds only the Debian family, so resolve-matrix.py rejects a Release entry - # for any other suite and this matrix is Debian by construction. + # fail-fast is disabled so a failed trixie release does not cancel the forky + # one; each entry is a package of its own and one failing does not make the + # others wrong to ship. # --------------------------------------------------------------------------- - build: + release: name: ${{ matrix.name }} needs: configure-matrix strategy: fail-fast: false matrix: - include: ${{ fromJson(needs.configure-matrix.outputs.matrix) }} + include: ${{ fromJson(needs.configure-matrix.outputs.releases) }} permissions: contents: read # actions/checkout in the called workflow packages: read # ghcr.io images it pulls - uses: ./.github/workflows/release-kernel-debian.yml + uses: ./.github/workflows/build-kernel-debian.yml with: suite: ${{ matrix.suite }} build: ${{ matrix.name }} @@ -102,7 +132,6 @@ jobs: git-clone: ${{ matrix.git_clone }} kernel-branch: ${{ matrix.branch_or_tag }} ref-strategy: ${{ matrix.ref_strategy }} - tag-pattern: ${{ matrix.tag_pattern || '' }} srcpkg: ${{ matrix.srcpkg }} binpkg: ${{ matrix.binpkg }} kernel-config: ${{ matrix.kernel_config }} @@ -111,8 +140,20 @@ jobs: localversion: ${{ matrix.localversion || '' }} kver-extra: ${{ matrix.kver_extra || '' }} debusine-parent-workspace: ${{ matrix.debusine_parent_workspace || '' }} + # Released archives only, as the nightly does. What this build resolves a + # Build-Depends against is something the released kernel will depend on, + # so reading qli-staging here would let a released kernel require a -dkms + # package that has not been released. + extra-build-dep-workspaces: qli + # The entry's own, and the one field that makes it a release rather than + # a build of the same ref: where the package is published. Stated in the + # matrix rather than defaulted here, so the archive a release reaches is + # reviewable alongside the ref it ships. target-workspace: ${{ matrix.target_workspace }} + # A release reaches an archive people install from, so the promotion + # waits for whatever approval the Production environment requires. The + # nightly leaves this at Staging and promotes unattended. + promote-environment: Production secrets: DEBUSINE_USER: ${{ secrets.DEBUSINE_USER }} DEBUSINE_TOKEN: ${{ secrets.DEBUSINE_TOKEN }} - DEBUSINE_RELEASE_TOKEN: ${{ secrets.DEBUSINE_RELEASE_TOKEN }} diff --git a/README.md b/README.md index 00117e0a..ff5b80a1 100644 --- a/README.md +++ b/README.md @@ -13,17 +13,17 @@ takes its packaging and its CI scripts from the same commit. The CI model is matrix-driven. This single repository can deliver multiple kernel variants, each with independent source/package identity, kernel source and ref strategy, configuration fragments, Debian revision, target suites, and -release destination. +archive destination. Every entry in the matrix is one isolated build leg producing one package. A -kernel flavour owns as many entries as it has suites and delivery types. +kernel flavour owns as many entries as it has suites. ### Configured variants -| Variant | Source package | Image metapackage | Daily suites | Release suites | Notes | -|---------|----------------|-------------------|--------------|----------------|-------| -| `qcom-next` | `linux-qcom-next` | `linux-image-qcom-next` | trixie, forky, resolute | trixie, forky | Standard kernel | -| `qcom-next-debug` | `linux-qcom-next-debug` | `linux-image-qcom-next-debug` | trixie, forky | trixie, forky | Adds `arch/arm64/configs/qcom_debug.config` and `kernel/configs/debug.config` from the kernel source, via `intree:` entries | +| Variant | Source package | Image metapackage | Suites | Promoted to | Notes | +|---------|----------------|-------------------|--------|-------------|-------| +| `qcom-next` | `linux-qcom-next` | `linux-image-qcom-next` | trixie, forky, resolute | `qli-staging` (trixie, forky) | Standard kernel | +| `qcom-next-debug` | `linux-qcom-next-debug` | `linux-image-qcom-next-debug` | trixie, forky | `qli-staging` | Adds `arch/arm64/configs/qcom_debug.config` and `kernel/configs/debug.config` from the kernel source, via `intree:` entries | | `qcom-arduino` | `linux-qcom-arduino` | `linux-image-qcom-arduino` | trixie, forky | none | Arduino hardware-enablement topic branch (`early/hwe/arduino` of `kernel-topics`) | | `mainline` | `linux-mainline` | `linux-image-mainline` | trixie, forky | none | Tip of Linus's tree, tracked for early warning of upstream breakage. No DKMS modules | | `next` | `linux-next` | `linux-image-next` | trixie, forky | none | Newest `next-YYYYMMDD` tag of linux-next. No DKMS modules | @@ -35,29 +35,35 @@ image package that can be installed alongside the others. The flavour is what the kernel is; a build's `name` is only what CI calls it. See [docs/version.md](docs/version.md) for how the version strings are composed. -The last three track a moving upstream — a branch tip, or a tag cut every -night — and so have no `Release` entries: a release must name an immutable ref -to promote. They are built and published daily and never promoted to `qli`. +The last three name no `target_workspace`: they track a moving upstream for +early warning, are published to S3, and never reach an archive anyone installs +from. `ci/build-matrix.yaml` is the source of truth; this table is a summary. -Two entry points use the same reusable build pipeline: +Entries come in two lists, each with a workflow that builds them: -- **Daily** uses the matrix-selected latest-tag or branch-tip strategy and - builds every configured Daily suite. -- **Release** uses a pinned matrix ref and promotes successful Debian packages - to the selected production Debusine workspace. +- **`daily.yml`** builds every `builds` entry nightly, using the matrix-selected + latest-tag or branch-tip strategy, and promotes each entry that names a + `target_workspace` into that workspace as part of the build that produced it. +- **`release.yml`** builds a `releases` entry on request, from the immutable ref + that entry pins, and promotes it into `qli` behind an approval. -One entry in `builds` is one generated package: a single `name`, a single -`type`, and a single `suite`. The `name` labels that one build and nothing -else — it is the Actions job name and what a manual dispatch asks for. Nothing -is expanded or derived at resolve time, so what an entry says is what gets -built: +Neither publishes a build some earlier run produced. The Debusine workspace a +build runs in is named after that run and does not outlive it, so the only +moment its contents can be published is while the run still holds it — +promotion is in the build or it is nowhere, and a release therefore builds the +ref it ships rather than promoting a nightly. What the archive carries is +therefore always an artifact the run that published it built and tested. + +One entry in `builds` is one generated package: a single `name` for a single +`suite`. The `name` labels that one build and nothing else — it is the Actions +job name and what a manual dispatch asks for. Nothing is expanded or derived at +resolve time, so what an entry says is what gets built: ```yaml builds: - name: qcom-next-trixie - type: Daily suite: trixie flavour: qcom-next git_clone: https://github.com/qualcomm-linux/kernel @@ -73,24 +79,7 @@ builds: - iris-vpu - audioreach debian_revision: '0qli1~bpo13+1~' - - - name: qcom-next-trixie - type: Release - suite: trixie - flavour: qcom-next - git_clone: https://github.com/qualcomm-linux/kernel - branch_or_tag: - ref_strategy: pinned_ref - srcpkg: linux-qcom-next - binpkg: linux-image-qcom-next - kernel_config: [] - dkms: - - kgsl - - camx - - iris-vpu - - audioreach - debian_revision: '0qli1~bpo13+1' - target_workspace: qli + target_workspace: qli-staging ``` Entries are written out in full rather than sharing YAML anchors, so each one @@ -100,25 +89,24 @@ consistency that duplication would otherwise put at risk — see Each entry states its `debian_revision` outright. The configured values are: -| Suite | Daily | Release | -| --- | --- | --- | -| Trixie | `0qli1~bpo13+1~` | `0qli1~bpo13+1` | -| Forky | `0qli1~` | `0qli1` | -| Resolute | `0qli1~26.04.1~` | (not a configured Release suite) | - -`~` always sorts below the same prefix without it in Debian version -ordering, so a suite's Daily always sorts below its Release; `resolve-matrix.py` -requires a Daily revision to be exactly its Release revision plus a trailing -`~`. Ordering across *different* suites depends entirely on the configured -revisions: with the values above, Resolute < Trixie < Forky for the same -delivery type, matching a Debian-backports-then-unstable promotion chain. -This is a deliberate ordering policy, not an automatic guarantee — adding a -suite means choosing a revision that sorts where that suite belongs relative to -the others. One nuance to be aware of: because Forky's revision carries no -suite component, its Daily revision ends immediately after the trailing `~`, so -Trixie Daily does not sort below Forky Daily even though Trixie Release sorts -below Forky Release. This does not affect the supported Release-to-Release -upgrade path. +| Suite | Revision | +| --- | --- | +| Trixie | `0qli~bpo13+1~` | +| Forky | `0qli~` | +| Resolute | `0qli~26.04.1~` | + +The trailing `~` marks the version as one nobody has blessed yet, and it stays +on the version through release. That is a consequence of releasing by +promotion: what reaches `qli` is the artifact `qli-staging` holds, byte for +byte, so its version is the version it was built with. A release that dropped +the `~` would have to be a different build, which is exactly what this design +removes. + +Ordering across *different* suites depends entirely on the configured +revisions: with the values above, Resolute < Trixie < Forky, matching a +Debian-backports-then-unstable promotion chain. This is a deliberate ordering +policy, not an automatic guarantee — adding a suite means choosing a revision +that sorts where that suite belongs relative to the others. `ci/build-matrix.yaml` is the authoritative configuration. Adding a kernel variant is a matrix change, not a workflow redesign. @@ -127,86 +115,139 @@ variant is a matrix change, not a workflow redesign. | Workflow | Purpose | Trigger | | --- | --- | --- | -| `daily.yml` | Resolves and runs the Daily matrix. The manual build entry point. | Scheduled daily at `23:00 UTC`, or manual dispatch. | -| `release.yml` | Resolves and runs the Release matrix. | Manual dispatch only. | -| `build-kernel-debian.yml` | Builds one Debian-suite entry in Debusine and publishes it to S3. | Called by Daily and PR build. | +| `daily.yml` | Resolves and runs the matrix. The manual build entry point. | Scheduled daily at `23:00 UTC`, or manual dispatch. | +| `release.yml` | Builds the pinned refs in the matrix's `releases` list and promotes them into the released archive. | Manual dispatch only. | +| `build-kernel-debian.yml` | Builds one Debian-suite entry in Debusine, publishes it to S3, and promotes it into the workspace its caller named, if any. | Called by Daily, Release and PR build. | | `build-kernel-ubuntu.yml` | Builds one Ubuntu-suite entry on the Docker path and publishes it to S3. | Called by Daily and PR build. | -| `release-kernel-debian.yml` | Builds one Debian-suite entry in Debusine and promotes it to the release workspace. | Called by Release. | -The three build workflows share their steps through two composite actions -rather than through a common orchestrator workflow: +The two build workflows share their steps through two composite actions rather +than through a common orchestrator workflow: | Action | Used by | | --- | --- | -| `.github/actions/prepare-kernel-source` | All three, as the `prepare` job. | -| `.github/actions/debusine-build` | The two Debian workflows, as the `build` job. | +| `.github/actions/prepare-kernel-source` | Both, as the `prepare` job. | +| `.github/actions/debusine-build` | The Debian workflow, as the `build` job. | Which of them a build leg calls is decided by the caller, from the entry's suite: `resolve-matrix.py --family debian|ubuntu` splits the selection, and each family's entries call only the workflow that builds them. Nothing inside a -build workflow is conditional on the suite or on whether the run releases, so a -run starts exactly the jobs it needs and shows no skipped job for a path it did -not take. +build workflow is conditional on the suite, so a run starts exactly the jobs it +needs and shows no skipped job for a path it did not take. ### Daily -Daily is the recurring build and artifact-publication path. +Daily builds every entry in the matrix, and everything in the staging archive +comes from a run of it. -- The scheduled run resolves the full `Daily` matrix. +- The scheduled run resolves the full matrix. - A manual run says which entries to build in one **Builds** field: - `all`, the default, builds every configured variant and suite. - A comma-separated list of build names builds those entries, e.g. - `qcom-next-trixie,qcom-next-debug-forky`. A name matching no `Daily` entry - fails the run rather than narrowing it, and both families are selected from - one list, so a mixed list starts Debian and Ubuntu legs from one dispatch. + `qcom-next-trixie,qcom-next-debug-forky`. A name matching no entry fails + the run rather than narrowing it, and both families are selected from one + list, so a mixed list starts Debian and Ubuntu legs from one dispatch. - `latest_tag` resolves the newest matching dated tag; `branch_tip` resolves the configured branch directly. - Debian suites build in Debusine, then their `.deb` outputs are downloaded and uploaded to the configured S3 bucket. +- An entry naming a `target_workspace` is then promoted into it with Debusine's + `package-publish` workflow, making it installable from that archive. - `resolute` stays on the Docker-based Ubuntu path and uploads its package outputs to the existing temporary-package S3 location. +Because the package version is a function of the resolved ref and the entry's +revision, a night on which the tracked tag has not moved would rebuild a +version the archive already has and then fail promoting the duplicate. The +`prepare` job therefore asks the target workspace whether it already holds the +version this run would produce, and skips the build, the S3 publication and the +promotion when it does. A run that skips this way is green: nothing was wrong, +there was simply nothing new upstream. + +Three kinds of run deliberately promote nowhere: a PR build, which never +forwards a `target_workspace`; a dispatch setting any of the advanced +validation overrides (`debug-build`, `qcom-next-pr`, `kernel-topics-pr`), +whose kernel is by definition not the one the matrix describes; and any entry +that names no workspace. + ### Release -Release is the controlled promotion path. - -- It is manual only, and says which entries to release in one **Builds** field, - the same way Daily does: - - `all`, the default, promotes every configured `Release` entry. - - A comma-separated list of build names promotes those entries when a - targeted action is required, e.g. `qcom-next-trixie,qcom-next-forky`. -- It uses the pinned `branch_or_tag` from the selected `Release` matrix row; it - never resolves a newest tag. -- Debian source and binary artifacts are built in per-flavour, per-suite - Debusine CI workspaces. -- Successful builds are promoted with Debusine's `package-publish` workflow to - the `qli` workspace, where they are available through the production Debusine - APT repository. -- The Release job runs in the **Production** GitHub environment. This provides - the release credential and enforces the required approval gate before - promotion to `qli`. - -A `daily.yml` dispatch is build-only: it publishes to the daily S3 path and has -no promotion path to offer. Release promotion is initiated exclusively by -`release.yml`, which owns the target workspace and production release controls, -and is the only caller of `release-kernel-debian.yml`. - -Only the Debian family has a release path at all, because promotion runs -through Debusine. `resolve-matrix.py` rejects a `Release` entry for any other -suite rather than letting it build and then silently not promote. +A release builds the ref it releases, and publishes the result into `qli`. + +- Every value describing a release lives in `ci/build-matrix.yaml` under + `releases`: the pinned ref, the suites, the packaging, and the + `target_workspace` each entry publishes into. A dispatch only says which + entries to run, in the same **Builds** field Daily uses. +- Updating a release is a pull request that changes `branch_or_tag`. That is + what puts the ref that ships under review, rather than a version typed into + a dispatch form at the moment of releasing. +- `ref_strategy` must be `pinned_ref`. `latest_tag` would make two dispatches + of one entry release different kernels, and `branch_tip` would make them + release whatever the branch had reached. +- Debian suites only, and `resolve-matrix.py` rejects anything else: promotion + runs through Debusine, and the Ubuntu path has no workspace to promote into. +- The `promote` job runs in the **Production** environment, so whatever + approval that environment requires stands between the run and the archive. +- Build-Depends resolve against `qli` alone, so a released kernel cannot depend + on a `-dkms` package that has not itself been released. + +It rebuilds rather than promoting a nightly because there is nothing left to +promote from: the CI workspace a nightly ran in is named after that run and does +not outlive it. What reaches `qli` is therefore the artifact this run built and +tested, from a ref that cannot have moved since it was reviewed. + +### Publishing + +Promotion happens inside the build, in the `promote` job of +`build-kernel-debian.yml`. + +- It reads the ephemeral CI child workspace the build ran in and publishes the + source and binary artifacts into the workspace the caller named. +- It is in-run by necessity. `lib/build` names that workspace + `-gh----`, creating it fresh for the + run and not keeping it afterwards, so a later workflow would have neither the + name nor the contents to promote. Publishing is part of the build or it does + not happen. +- Only the Debian family reaches it, because promotion runs through Debusine + and the Ubuntu path does not build there. +- Which GitHub environment it runs in is the caller's, through + `promote-environment`. Daily leaves it at **Staging** and promotes + unattended, because `qli-staging` is where an unreviewed nightly kernel + belongs; Release passes **Production**, so an approval stands in front of + `qli`. + +A nightly promotion changes no file in this repository: there is no ref to pin +and no entry to bless, because what is published is what was built. A release +does, and that is the difference between them — the ref it ships is written +down and reviewed before the run that ships it. + +#### Moving the nightly to `qli` + +`qli-staging` is the nightly's destination, named by the `target_workspace` of +each nightly entry in [ci/build-matrix.yaml](ci/build-matrix.yaml). Pointing +those at `qli` would publish every night's kernel straight into the released +archive with nothing in between, and the approval gate that +`promote-environment` provides would then have to be applied to every nightly +run — stopping each of them to wait for one. The two archives, with +`release.yml` between them, are what keep nightlies unattended and `qli` gated. ## Matrix Model -`ci/build-matrix.yaml` is a mapping with exactly one top-level key, `builds`. -One entry in it is one generated package, so there is no +`ci/build-matrix.yaml` is a mapping with two top-level keys, `builds` and +`releases`. One entry in either is one generated package, so there is no expansion step: `ci/scripts/resolve-matrix.py` validates the whole document, -selects the entries matching the requested type, variant, and suite, and hands -them to the workflow matrix as they stand. Each entry carries: +selects the entries matching the requested names, variant, and suite, and hands +them to the workflow matrix as they stand. + +The two lists have the same shape and are validated separately, so a name may +appear in both and means the same build in each. `builds` is what Daily runs +nightly; `releases` is what `release.yml` can ship, selected with +`--releases`. Both are validated on every invocation whichever is asked for, so +a broken release entry fails a nightly run rather than waiting to be found by +whoever next tries to release. Each entry carries: | Field | Purpose | | --- | --- | -| `name` | The name of this one build, and nothing else: its Actions job name, and what a manual dispatch asks for. Unique within a delivery type, so a Daily and its Release may share a name. Never reaches a package name, a version, or a published path. Lowercase letters, digits, and internal hyphens only, and not `all`, which a dispatch reads as every entry. | -| `type` | `Daily` or `Release`. | +| `name` | The name of this one build, and nothing else: its Actions job name, and what a manual dispatch asks for. Unique across the matrix. Never reaches a package name, a version, or a published path. Lowercase letters, digits, and internal hyphens only, and not `all`, which a dispatch reads as every entry. | | `suite` | The one suite this entry builds for. | | `flavour` | The kernel's own identity, and the only matrix field that reaches the built kernel. `derive-localversion.sh` makes it the LOCALVERSION suffix, so two flavours built from one ref get distinct kernel releases and their `linux-image` packages coexist. All entries sharing a flavour build the same package for different suites. | | `git_clone` | Kernel source repository. | @@ -217,50 +258,45 @@ them to the workflow matrix as they stand. Each entry carries: | `binpkg` | Kernel image metapackage name. | | `kernel_config` | Extra fragments applied on top of `debian/config-available/`, all of which is applied to every build, one per list element. A bare name selects `debian/config-available/.config`; an `intree:` entry names a fragment shipped by the kernel source, as a path relative to the kernel source root (e.g. `intree:arch/arm64/configs/qcom_debug.config`), so it stays versioned with the kernel it targets. Empty for variants that need nothing beyond `config-available/`; today it carries only `intree:` fragments. `resolve-matrix.py` joins it into the comma-separated `kernel-config` workflow input. | | `dkms` | Out-of-tree DKMS modules built and bundled into the image package, one per list element, each named without the `-dkms` suffix (e.g. `kgsl`). Each needs a `-dkms` package in the suite being built for, so this varies between suites. An empty list bundles nothing. A listed module is a presence contract: a build fails rather than shipping an image without it. `resolve-matrix.py` joins it into the comma-separated `dkms` workflow input, which reaches `prepare-source.sh --dkms`; see [debian/README.md](debian/README.md) for what the packaging does with it. | -| `debian_revision` | The Debian revision this package is built at, stated outright. Daily revisions end in `~`; Release revisions do not. | +| `debian_revision` | The Debian revision this package is built at, stated outright. Carried into every archive the package reaches, because a release promotes the built artifact rather than rebuilding it. | | `localversion`, `kver_extra` | Optional version overrides forwarded to packaging. | | `debusine_parent_workspace` | Optional parent workspace override for the variant's CI child workspaces. | -| `target_workspace` | Debusine destination for Release entries only. | +| `target_workspace` | The Debusine workspace this entry publishes into. On a `builds` entry it is optional and names where the nightly promotes it; an entry without one is built and published to S3 and reaches no archive. On a `releases` entry it is required, because putting one ref into one archive is the whole of what a release is. Debian suites only. | `resolve-matrix.py` rejects the matrix — before any build job starts — where an entry has an unknown field or a missing required one, a malformed variant or -suite identifier, a `ref_strategy` its `type` does not allow (`Daily` must -track something moving, `Release` must be pinned), a `tag_pattern` without -`latest_tag`, a `target_workspace` on a `Daily` entry or none on a `Release` -one, a `kernel_config` fragment that escapes the kernel source root or collides -with another fragment's filename, a `dkms` entry that is not a package name -stem or repeats, or a `debian_revision` that is not a valid Debian revision or -carries the wrong trailing `~` for its `type`. +suite identifier, an unknown `ref_strategy`, a `tag_pattern` without +`latest_tag`, a `target_workspace` on a non-Debian suite, a `kernel_config` +fragment that escapes the kernel source root or collides with another +fragment's filename, a `dkms` entry that is not a package name stem or repeats, +or a `debian_revision` that is not a valid Debian revision. + +A `releases` entry is held to two rules of its own: `ref_strategy` must be +`pinned_ref`, so releasing twice releases the same kernel, and `suite` must be +a Debian one, because only that path can promote into a workspace. Because entries are written out in full, the resolver also checks the invariants that span them, which is what makes the duplication safe to read at face value: -- No two entries of one `type` share a `name` — a name identifies exactly one - build, so a run cannot produce two jobs with one name and a dispatch cannot - be ambiguous. +- No two entries share a `name` — a name identifies exactly one build, so a run + cannot produce two jobs with one name and a dispatch cannot be ambiguous. - A flavour's entries agree on `srcpkg`, `binpkg` and `kernel_config`; those decide what the package *is*, and the entries differ only in where it goes. - A flavour's entries for one suite agree on `dkms`. The module set depends on which `-dkms` packages the target archive carries, so it varies between - suites — but a suite's Daily and Release must match, or the Daily is not - testing the module set the Release will ship. -- A flavour's entries of one `type` agree on `git_clone`, `branch_or_tag`, - `ref_strategy` and `tag_pattern`, so a forgotten suite cannot quietly ship a - different kernel from its siblings after a release ref bump. -- A flavour with a `Release` entry also has a `Daily` one, so nothing is - promoted that the daily build has not tested. The converse is allowed: a - flavour tracking a moving upstream is built daily and never released. + suites but not within one. +- A flavour's entries agree on `git_clone`, `branch_or_tag`, `ref_strategy` and + `tag_pattern`, so a forgotten suite cannot quietly ship a different kernel + from its siblings. - No `srcpkg` or `binpkg` is shared between flavours, and no two entries build the same `srcpkg` at the same `debian_revision`. -- Where a suite has both, its `Daily` revision is its `Release` revision plus a - trailing `~`. The build workflows pass their entry's own `debian_revision` through. A caller with no entry in hand can leave the `debian-revision` input empty, and the -`prepare-kernel-source` action looks up the `Daily` entry for the build name and -suite it was given (`resolve-matrix.py --field debian_revision`) and builds at -the revision the daily build would have used. +`prepare-kernel-source` action looks up the entry for the build name and suite +it was given (`resolve-matrix.py --field debian_revision`) and builds at the +revision the nightly build uses. Each entry has a distinct prepared-source artifact, Debusine child workspace, and S3 path keyed by `flavour + suite`. This prevents two flavours that both @@ -268,7 +304,7 @@ build, for example, `trixie` from consuming or publishing each other's inputs or outputs. They key on `flavour`, not on the build's `name`, so renaming a build never moves a published artifact. -Daily S3 outputs use these layouts, where `` is +S3 outputs use these layouts, where `` is `-`: ```text @@ -298,23 +334,25 @@ This document covers the CI generator. For the packaging internals: `debian/rule targets, the config fragment merge pipeline, DKMS module bundling and the produced package layout see [debian/README.md](debian/README.md). -Both branches below are taken in the caller, when the matrix is resolved: the -family from the entry's suite, the tail from which workflow is running. By the -time a build workflow starts, there is nothing left to decide. +The family branch below is taken in the caller, when the matrix is resolved, +from the entry's suite. The promotion branch is taken from the entry's own +`target_workspace`. By the time a build workflow starts, there is nothing left +to decide. ```mermaid flowchart LR IN["Matrix entries"] --> R{"resolve-matrix.py\n--family"} - R -->|"debian: trixie · forky"| DT{"Which caller"} + R -->|"debian: trixie · forky"| DEB["build-kernel-debian.yml\nGenerate source package\nSubmit with lib/build\nDebusine builds binaries"] R -->|"ubuntu: resolute"| UBU["build-kernel-ubuntu.yml\nbuild-kernel.sh in Docker\nBuild binary packages"] - DT -->|"daily.yml · pr-build.yml"| DEB["build-kernel-debian.yml\nGenerate source package\nSubmit with lib/build\nDebusine builds binaries"] - DT -->|"release.yml"| REL["release-kernel-debian.yml\nSame build, release tail"] - DEB --> S3["Download .deb files\nPublish to S3"] - REL --> QLI["Promote source and binaries\nto qli"] + DEB --> TW{"target_workspace\nset?"} + TW -->|"yes, via daily.yml"| STG["Promote to qli-staging\nStaging environment"] + TW -->|"yes, via release.yml"| REL["Promote to qli\nProduction environment"] + TW -->|no| NONE["No archive"] UBU --> US3["Publish .deb files to S3"] + ``` ## For CI Maintainers @@ -326,56 +364,57 @@ flowchart TD subgraph triggers[Triggers] A1["daily.yml\nScheduled full matrix"] A2["daily.yml\nManual: all or named builds"] - A5["pr-build.yml\nFull Daily matrix on every PR"] - A3["release.yml\nManual: all or named builds"] + A3["release.yml\nManual: pinned refs from releases"] + A5["pr-build.yml\nFull matrix on every PR"] + A3["release.yml\nManual: version to release"] end subgraph matrix[Matrix entry points] - B1["configure-matrix\nDaily entries, split by family"] + B1["configure-matrix\nEntries, split by family"] B2["build-debian legs\nqcom-next · qcom-next-debug · qcom-arduino\nmainline · next / trixie · forky"] B5["build-ubuntu legs\nqcom-next / resolute"] - B3["configure-matrix\nRelease entries, Debian by construction"] - B4["build legs\nqcom-next · qcom-next-debug / trixie · forky"] + B3["configure-matrix\nEntries with a target_workspace"] end subgraph build[One build workflow per leg] - C2["prepare\nprepare-kernel-source action\nClone ref, run prepare-source.sh\nUpload kernel-srcpkg-flavour-suite"] + C2["prepare\nprepare-kernel-source action\nClone ref, run prepare-source.sh\nSkip the run if the version is published"] C3["build\ndebusine-build action"] C4["build\nbuild-kernel.sh in Docker"] C5["publish\nDownload .deb files, upload to S3"] - C6["release\nPromote to target workspace"] + C6["promote\nlib/release into target_workspace"] end subgraph outputs[Outputs] - D1["Daily S3 artifacts"] - D2["Release qli APT repository"] + D1["S3 artifacts"] + D3["qli-staging APT repository"] + D4["qli APT repository"] end A1 --> B1 A2 --> B1 + A3 --> B1 A5 --> B1 - A3 --> B3 B1 --> B2 & B5 - B3 --> B4 B2 --> C2 B5 --> C2 - B4 --> C2 C2 --> C3 & C4 C3 --> C5 & C6 C4 --> D1 C5 --> D1 - C6 --> D2 + C6 --> D3 & D4 ``` -Every leg runs every job drawn under it. `build-debian` and `build-ubuntu` legs -reach different build workflows, and `publish` and `release` belong to -different ones, so no leg starts a job it will skip. +A leg runs every job drawn under it except `promote`, which only a caller +naming a workspace reaches — so a PR build stops at S3. Which archive `promote` +reaches, and whether it waits for an approval first, is that caller's too: the +nightly goes to `qli-staging` unattended, and a release to `qli` through the +Production environment. ### Prepare stage ```mermaid flowchart LR - K["Matrix-selected kernel repository\nDaily: latest tag or branch tip\nRelease: pinned ref"] --> PS + K["Matrix-selected kernel repository\nlatest tag, branch tip, or pinned ref"] --> PS M["pkg-linux-qcom\ndebian/ and ci/ from this commit"] --> PS PS["prepare-source.sh\n\nInject debian/\nApply all config-available fragments plus any extras\nGenerate control, changelog, localversion, pkgversion"] --> TAR @@ -387,7 +426,7 @@ flowchart LR > Unix execute bits. Kernel build scripts require those permissions. The tar > archive preserves them between the prepare and build jobs. -### Debian Daily path +### Debian build path ```mermaid flowchart LR @@ -401,23 +440,28 @@ flowchart LR SUBMIT --> DEB["Debusine\nBuild binary packages"] DEB --> WS["Unique variant + suite workspace"] - subgraph publish[Daily publish job] + subgraph publish[publish job] WS --> APT["generate-apt-config\nchdist isolated APT environment\nDownload .deb files"] - APT --> S3["S3\nDaily package artifacts"] + APT --> S3["S3\npackage artifacts"] + end + + subgraph promote[promote job: only when target_workspace is set] + WS --> PROMOTE["lib/release\nStart package-publish"] + PROMOTE --> STG["qli-staging (daily)\nqli (release)\nDebusine APT repository"] end ``` -### Debian Release path +### Release path + +Nothing is built. `release.yml` promotes a version that the nightly build +already put in `qli-staging`. ```mermaid flowchart LR - ART["kernel-srcpkg-variant-suite\nartifact"] --> GSP["generate-source-package\nProduce .dsc"] - GSP --> SUBMIT["lib/build\nSubmit source package to a unique\nDebusine CI child workspace"] - SUBMIT --> DEB["Debusine\nBuild binary packages"] - DEB --> WS["CI workspace\nsource and binary artifacts"] + STG["qli-staging\nsource and binary artifacts"] --> PROMOTE - subgraph release[Release job: Production GitHub environment] - WS --> PROMOTE["lib/release\nStart package-publish"] + subgraph release[promote job: Production GitHub environment] + PROMOTE["lib/release\nSRCPKG_VERSION = upstream-version + entry revision\nStart package-publish"] PROMOTE --> QLI["qli\nProduction Debusine APT repository"] end ``` @@ -433,7 +477,7 @@ flowchart LR BK["build-kernel.sh\n--skip-prepare\n--local-source\n--build-mode docker\ndpkg-buildpackage -b"] --> S3 end - S3["S3\nDaily package artifacts"] + S3["S3\npackage artifacts"] ``` `--skip-prepare` is safe because `prepare-source.sh` has already generated the @@ -490,21 +534,29 @@ out-of-tree module builds are required. ## Manual Builds Use **Actions** → **daily** → **Run workflow** for a one-off build, and -**Actions** → **release** to promote. Both are dispatched the same way: one -**Builds** field naming what to run. +**Actions** → **release** to promote a version that has already been built. +Both name what to act on in one **Builds** field. | Input | Default | Purpose | | --- | --- | --- | -| `builds` | `all` | `all` runs every entry of that workflow's delivery type. Otherwise a comma-separated list of build `name` values from `ci/build-matrix.yaml`, e.g. `qcom-next-trixie,qcom-next-debug-forky`. | +| `builds` | `all` | `all` selects every entry — for `release`, every entry that names a `target_workspace`. Otherwise a comma-separated list of build `name` values from `ci/build-matrix.yaml`, e.g. `qcom-next-trixie,qcom-next-debug-forky`. | Everything else about a build — its suite, flavour, kernel repository and ref, package names, config fragments, DKMS modules and Debian revision — comes from the entry, so there is nothing to retype and nothing to get wrong. Build names are free-text matrix values rather than a static dropdown, so adding a matrix entry never requires editing the workflow UI, and a name that matches no entry -of the delivery type fails the run with the list of names that do. `daily` -routes each selected entry to the workflow that builds its family, so one -dispatch can name Debian and Ubuntu builds together. +fails the run with the list of names that do. `daily` routes each selected +entry to the workflow that builds its family, so one dispatch can name Debian +and Ubuntu builds together. + +`release` carries two further inputs, because a promotion has to say what it is +promoting: + +| Input | Default | Purpose | +| --- | --- | --- | +| `upstream-version` | None, required | The version to release without its Debian revision, e.g. `7.2.0~rc7+20260821`. Each selected entry promotes this plus its own `debian_revision`. | +| `release-workspace` | `qli` | The Debusine workspace to promote into. | `daily` carries three further inputs, which the matrix deliberately says nothing about because they belong to a one-off validation run rather than to a @@ -517,12 +569,28 @@ them at their defaults: | `qcom-next-pr` | Empty | Advanced Qualcomm-only override: `qcom-next` PR numbers to merge before building. | | `kernel-topics-pr` | Empty | Advanced Qualcomm-only override: `kernel-topics` PR numbers to apply as patches. | -A `daily` dispatch is an artifact build and publishes to the daily S3 path; -Release promotion is performed only through `release.yml`. The build workflows -themselves (`build-kernel-debian.yml`, `build-kernel-ubuntu.yml`, -`release-kernel-debian.yml`) are `workflow_call` only and cannot be dispatched: -one run of each is one matrix entry, and a reusable workflow cannot fan itself -out over a list. +A `daily` dispatch publishes to S3 and, for an entry that names one, promotes +into that entry's `target_workspace` — unless it sets one of the advanced +overrides above, which suppress promotion because the kernel they build is not +the one the matrix describes. Promotion into the release workspace happens only +through `release.yml`. The build workflows themselves +(`build-kernel-debian.yml`, `build-kernel-ubuntu.yml`) are `workflow_call` only +and cannot be dispatched: one run of each is one matrix entry, and a reusable +workflow cannot fan itself out over a list. + +### Releasing + +Use **Actions** → **release** → **Run workflow**. It takes the same **Builds** +field, selecting from the matrix's `releases` list rather than from `builds`, +and nothing else — the ref, the packaging and the destination archive are all +the entry's. + +Release a kernel by opening a pull request that sets `branch_or_tag` on the +entries being released, merging it, and dispatching `release` for those names. +The run stops at the `promote` job for the Production environment's approval, +having already built, published to S3 and printed what it is about to release +in the run summary, so the approval is given against the refs and versions in +front of you. ## Configuration @@ -530,7 +598,7 @@ out over a list. | Variable | Purpose | | --- | --- | -| `ARTIFACT_S3_BUCKET` | S3 bucket for Daily Debian artifacts and Ubuntu build artifacts. | +| `ARTIFACT_S3_BUCKET` | S3 bucket for Debian and Ubuntu build artifacts. | | `DEBUSINE_HOST` | Production Debusine host. | | `DEBUSINE_SCOPE` | Debusine scope. | | `DEBUSINE_PARENT_WORKSPACE` | Parent workspace used to create per-run CI child workspaces. | @@ -540,54 +608,67 @@ out over a list. | Secret | Scope | Purpose | | --- | --- | --- | | `DEBUSINE_USER` | Repository | User for Debusine archive and signing-key access. | -| `DEBUSINE_TOKEN` | Repository | Token for Debusine build and artifact operations. | -| `DEBUSINE_RELEASE_TOKEN` | Production environment | Token used only to promote Release artifacts to `qli`. | +| `DEBUSINE_TOKEN` | Repository | Token for Debusine build and artifact operations, including the nightly promotion into `qli-staging`. | +| `DEBUSINE_RELEASE_TOKEN` | Production environment | Token used only to promote into the release workspace. | -The Debian build and Release jobs select the **Production** GitHub environment. -This makes environment-scoped release credentials available to the promotion job -and keeps production approval controls in the workflow path. +The `build` job of `build-kernel-debian.yml` always selects the **Staging** +GitHub environment. The `promote` job selects whichever its caller names +through `promote-environment`: **Staging** for the nightly, which is what lets +it promote unattended, and **Production** for a release, which is what makes it +wait for an approval. + +`DEBUSINE_TOKEN` is what both promotions authenticate with, so it needs write +access to every `target_workspace` the matrix names — `qli-staging` for the +nightly entries and `qli` for the `releases` ones. `DEBUSINE_RELEASE_TOKEN` is not read by +any workflow; a release is gated by the Production environment rather than by a +credential of its own. ## Maintaining the Matrix To add a kernel variant: 1. Add one entry to `builds` per package the variant should produce: one per - Daily suite and one per Release suite, each spelling out all of its own - fields, and each with a `name` unique within its delivery type. Do not rely - on another variant's values. + suite, each spelling out all of its own fields, and each with a `name` + unique across the matrix. Do not rely on another variant's values. 2. Give them all the same `flavour`, distinct from every other flavour's — it becomes the LOCALVERSION suffix, so this is what lets the new kernel install - alongside the existing ones. Keep `srcpkg`, `binpkg` and `kernel_config` - identical across every entry for the flavour, and the ref fields identical - across its entries of one `type`. `resolve-matrix.py` rejects the matrix if - they drift apart. -3. Use `latest_tag` with a dated tag glob or `branch_tip` for Daily. Use - `pinned_ref` for Release, and update that ref through a reviewed PR. + alongside the existing ones. Keep `srcpkg`, `binpkg`, `kernel_config` and + the ref fields identical across every entry for the flavour. + `resolve-matrix.py` rejects the matrix if they drift apart. +3. Use `latest_tag` with a dated tag glob, or `branch_tip`, to track a moving + upstream; `pinned_ref` freezes the variant on one ref. 4. Give the variant distinct `srcpkg` and `binpkg` values. Set - `target_workspace` on each Release entry. -5. Give each entry a `debian_revision`: the Daily one is the Release one for - the same suite plus a trailing `~`. + `target_workspace` on the Debian entries that should be installable from an + archive, and leave it off the ones built only for early warning. +5. Give each entry a `debian_revision` that sorts where its suite belongs + relative to the others and collides with no other entry building the same + `srcpkg`. 6. Confirm suite-family routing: Debian suites use Debusine; Ubuntu suites use the Docker path. -7. Run a filtered Daily validation for the new variant, then its full Daily and - Release flows. +7. Run a filtered daily validation for the new variant, then a full daily run. + Nothing further is needed to release it: once its packages are in the + staging workspace, `release.yml` can promote them. To add a new suite (for an existing or new variant): -1. Add one entry per delivery type the suite should get, copying the variant's - existing entry for that type and changing `suite` and `debian_revision`. +1. Add one entry for the suite, copying one of the variant's existing entries + and changing `suite` and `debian_revision`. 2. Choose the revision so the suite sorts where it belongs relative to the - others for the same delivery type (see the ordering discussion in - [Overview](#overview)), and so it does not collide with another entry - building the same `srcpkg`. + others (see the ordering discussion in [Overview](#overview)), and so it + does not collide with another entry building the same `srcpkg`. + +To make a variant releasable, add the matching entries to `releases`: the same +fields, plus `target_workspace`, with `ref_strategy: pinned_ref` and +`branch_or_tag` naming the ref that ships. A variant with no `releases` entries +is built nightly and never shipped, which is the right state for a topic branch +or a tracking build. -No workflow dispatch choices need to be updated: the Daily and Release +No workflow dispatch choices need to be updated: the daily and release dispatches take build names as free text, so a new entry is dispatchable by name, and is picked up by `all`, as soon as it is merged. -Run `ci/scripts/resolve-matrix.py --type Daily` and `--type Release` locally to -validate a matrix change before pushing it; both validate the whole document, -so either one catches a mistake in the other's entries. +Run `ci/scripts/resolve-matrix.py` locally to validate a matrix change before +pushing it; it validates the whole document regardless of what it selects. ## Contributing diff --git a/ci/build-matrix.yaml b/ci/build-matrix.yaml index 00d72d61..56f85e09 100644 --- a/ci/build-matrix.yaml +++ b/ci/build-matrix.yaml @@ -3,19 +3,18 @@ # # Kernel delivery matrix. # -# One entry in "builds" is one generated package: a single name, a single -# delivery type, and a single suite. The only list-valued fields are -# kernel_config and dkms, which name the fragments and the out-of-tree modules -# that go into that one package, so an entry needs no expansion — ci/scripts/ -# resolve-matrix.py validates the document, filters it, and hands the matching -# entries straight to the workflow matrix. +# One entry in "builds" is one generated package: a single name for a single +# suite. The only list-valued fields are kernel_config and dkms, which name the +# fragments and the out-of-tree modules that go into that one package, so an +# entry needs no expansion — ci/scripts/resolve-matrix.py validates the +# document, filters it, and hands the matching entries straight to the workflow +# matrix. # # Entries are written out in full rather than sharing YAML anchors, so each one # can be read, grepped, and changed on its own. resolve-matrix.py enforces the # consistency that duplication would otherwise put at risk: a flavour's entries -# must agree on srcpkg, binpkg and kernel_config, its entries of one delivery -# type must agree on the kernel ref they build, and its entries for one suite -# must agree on dkms. +# must agree on srcpkg, binpkg, kernel_config and the kernel ref they build, +# and its entries for one suite must agree on dkms. # # The two identifiers are not the same thing. flavour is the kernel's own # identity: it becomes the LOCALVERSION suffix, so qcom-next and @@ -23,31 +22,46 @@ # other, and every suite building one flavour produces the same kernel release. # A build's name is what CI calls it: the Actions job name, and what a manual # dispatch asks for by name. It never reaches a package name or version, and -# never a published path. It must be unique within a delivery type, so a Daily -# and the Release that supersedes it share one. It may not be "all", which the -# daily and release dispatch forms read as every entry of their type. +# never a published path. It must be unique, and may not be "all", which the +# daily and release dispatch forms read as every entry. # # dkms varies by suite because it depends on which -dkms packages the # target archive carries: the Debian suites bundle camx and iris-vpu, which # Ubuntu does not package. # -# A flavour tracking a moving upstream is Daily only. A Release entry must name -# an immutable ref, and a branch tip or a tag cut every night is not one, so -# mainline, next and the topic branches are built and published daily and never -# promoted. A Release entry still requires the Daily that tests it. +# Every entry under "builds" is built nightly. An entry that names a +# target_workspace is also promoted into it once the build succeeds, so the +# archive carries the artifact the nightly build tested rather than a rebuild +# of the same ref. mainline, next and the topic branches name no +# target_workspace: they are early-warning builds, published to S3 and nowhere +# else. # -# debian_revision is stated outright, not derived. Daily revisions end in "~" -# so they sort below the Release revision for the same suite; the rest of the -# revision places the suite where it belongs relative to the other suites (see -# the ordering discussion in README.md). No two entries may produce the same -# srcpkg at the same debian_revision. +# "releases" is the other list, and the one place a shipped ref is written +# down. A release is a decision rather than a schedule: nothing runs these +# entries until someone dispatches release.yml, and what they name is an +# immutable ref rather than whatever the branch has reached. Each states the +# workspace it publishes into -- so the ref that ships and the archive it ships +# into are both reviewable in a pull request, rather than typed into a dispatch +# form at the moment of releasing. +# +# The two lists are validated separately and never mix: a name may appear in +# both, and means the same build in each. +# +# Only a Debian suite may name a target_workspace. Promotion runs through +# Debusine, which does not build the Ubuntu family. +# +# debian_revision is stated outright, not derived. The trailing "~" marks a +# version as unreleased, and it is carried into the archive as built: what +# release.yml promotes is the tested artifact, not a rebuild at a tidier +# version. The rest of the revision places the suite where it belongs relative +# to the other suites (see the ordering discussion in README.md). No two +# entries may produce the same srcpkg at the same debian_revision. builds: # --------------------------------------------------------------------------- # qcom-next — the standard kernel. # --------------------------------------------------------------------------- - name: qcom-next-trixie - type: Daily suite: trixie flavour: qcom-next git_clone: https://github.com/qualcomm-linux/kernel @@ -63,9 +77,9 @@ builds: - iris-vpu - audioreach debian_revision: '0qli1~bpo13+1~' + target_workspace: qli-staging - name: qcom-next-forky - type: Daily suite: forky flavour: qcom-next git_clone: https://github.com/qualcomm-linux/kernel @@ -81,9 +95,9 @@ builds: - iris-vpu - audioreach debian_revision: '0qli1~' + target_workspace: qli-staging - name: qcom-next-resolute - type: Daily suite: resolute flavour: qcom-next git_clone: https://github.com/qualcomm-linux/kernel @@ -98,48 +112,11 @@ builds: - kgsl debian_revision: '0qli1~26.04.1~' - - name: qcom-next-trixie - type: Release - suite: trixie - flavour: qcom-next - git_clone: https://github.com/qualcomm-linux/kernel - branch_or_tag: qcom-next-7.2-20260826 - ref_strategy: pinned_ref - srcpkg: linux-qcom-next - binpkg: linux-image-qcom-next - kernel_config: [] - dkms: - - kgsl - - camx - - iris-vpu - - audioreach - debian_revision: '0qli1~bpo13+1' - target_workspace: qli - - - name: qcom-next-forky - type: Release - suite: forky - flavour: qcom-next - git_clone: https://github.com/qualcomm-linux/kernel - branch_or_tag: qcom-next-7.2-20260826 - ref_strategy: pinned_ref - srcpkg: linux-qcom-next - binpkg: linux-image-qcom-next - kernel_config: [] - dkms: - - kgsl - - camx - - iris-vpu - - audioreach - debian_revision: '0qli1' - target_workspace: qli - # --------------------------------------------------------------------------- # qcom-next-debug — the same kernel ref built with the in-tree debug # fragments, packaged separately so it installs alongside qcom-next. # --------------------------------------------------------------------------- - name: qcom-next-debug-trixie - type: Daily suite: trixie flavour: qcom-next-debug git_clone: https://github.com/qualcomm-linux/kernel @@ -157,9 +134,9 @@ builds: - iris-vpu - audioreach debian_revision: '0qli1~bpo13+1~' + target_workspace: qli-staging - name: qcom-next-debug-forky - type: Daily suite: forky flavour: qcom-next-debug git_clone: https://github.com/qualcomm-linux/kernel @@ -177,54 +154,15 @@ builds: - iris-vpu - audioreach debian_revision: '0qli1~' - - - name: qcom-next-debug-trixie - type: Release - suite: trixie - flavour: qcom-next-debug - git_clone: https://github.com/qualcomm-linux/kernel - branch_or_tag: qcom-next-7.2-20260826 - ref_strategy: pinned_ref - srcpkg: linux-qcom-next-debug - binpkg: linux-image-qcom-next-debug - kernel_config: - - intree:arch/arm64/configs/qcom_debug.config - - intree:kernel/configs/debug.config - dkms: - - kgsl - - camx - - iris-vpu - - audioreach - debian_revision: '0qli1~bpo13+1' - target_workspace: qli - - - name: qcom-next-debug-forky - type: Release - suite: forky - flavour: qcom-next-debug - git_clone: https://github.com/qualcomm-linux/kernel - branch_or_tag: qcom-next-7.2-20260826 - ref_strategy: pinned_ref - srcpkg: linux-qcom-next-debug - binpkg: linux-image-qcom-next-debug - kernel_config: - - intree:arch/arm64/configs/qcom_debug.config - - intree:kernel/configs/debug.config - dkms: - - kgsl - - camx - - iris-vpu - - audioreach - debian_revision: '0qli1' - target_workspace: qli + target_workspace: qli-staging # --------------------------------------------------------------------------- # qcom-arduino — the Arduino hardware-enablement topic branch. # - # Daily only: a topic branch has no immutable ref to promote from. + # No target_workspace: a topic branch is built for early warning, not for + # anyone to install from an archive. # --------------------------------------------------------------------------- - name: qcom-arduino-trixie - type: Daily suite: trixie flavour: qcom-arduino git_clone: https://github.com/qualcomm-linux/kernel-topics @@ -240,7 +178,6 @@ builds: debian_revision: '0qli~bpo13+1~' - name: qcom-arduino-forky - type: Daily suite: forky flavour: qcom-arduino git_clone: https://github.com/qualcomm-linux/kernel-topics @@ -258,11 +195,11 @@ builds: # --------------------------------------------------------------------------- # mainline — Linus's tree, tracked for early warning of upstream breakage. # - # Daily only, and no DKMS: the out-of-tree modules target the Qualcomm tree, - # and a listed module is a presence contract that would fail the build here. + # No target_workspace, and no DKMS: the out-of-tree modules target the + # Qualcomm tree, and a listed module is a presence contract that would fail + # the build here. # --------------------------------------------------------------------------- - name: mainline-trixie - type: Daily suite: trixie flavour: mainline git_clone: https://github.com/torvalds/linux @@ -275,7 +212,6 @@ builds: debian_revision: '0qli~bpo13+1~' - name: mainline-forky - type: Daily suite: forky flavour: mainline git_clone: https://github.com/torvalds/linux @@ -291,10 +227,10 @@ builds: # next — linux-next, the integration tree, one step ahead of mainline. # # next-YYYYMMDD tags are cut daily, so latest_tag resolves the newest of - # them the same way qcom-next-* does. Daily only, and no DKMS, as above. + # them the same way qcom-next-* does. No target_workspace and no DKMS, as + # above. # --------------------------------------------------------------------------- - name: next-trixie - type: Daily suite: trixie flavour: next git_clone: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git @@ -308,7 +244,6 @@ builds: debian_revision: '0qli~bpo13+1~' - name: next-forky - type: Daily suite: forky flavour: next git_clone: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git @@ -320,3 +255,96 @@ builds: kernel_config: [] dkms: [] debian_revision: '0qli~' + +# Releases. Dispatched from release.yml and never scheduled, so nothing here +# runs until someone asks for it by name. +# +# Every entry pins its ref: ref_strategy must be pinned_ref, because a release +# names the exact source that ships. latest_tag would make two dispatches of +# one entry release different kernels, and branch_tip would make them release +# whatever the branch had reached. +# +# Debian suites only. Promotion runs through Debusine, and the Ubuntu path +# publishes to S3 without a workspace to promote into, so an Ubuntu entry here +# would name a destination nothing could deliver it to. +# +# Updating a release means editing branch_or_tag in a pull request, which is +# what puts the ref that ships under review. +releases: + # --------------------------------------------------------------------------- + # qcom-next 7.2 — the standard kernel. + # --------------------------------------------------------------------------- + - name: qcom-next-trixie + suite: trixie + flavour: qcom-next + git_clone: https://github.com/qualcomm-linux/kernel + branch_or_tag: qcom-next-7.2-20260826 + ref_strategy: pinned_ref + srcpkg: linux-qcom-next + binpkg: linux-image-qcom-next + kernel_config: [] + dkms: + - kgsl + - camx + - iris-vpu + - audioreach + debian_revision: '0qli1~bpo13+1' + target_workspace: qli + + - name: qcom-next-forky + suite: forky + flavour: qcom-next + git_clone: https://github.com/qualcomm-linux/kernel + branch_or_tag: qcom-next-7.2-20260826 + ref_strategy: pinned_ref + srcpkg: linux-qcom-next + binpkg: linux-image-qcom-next + kernel_config: [] + dkms: + - kgsl + - camx + - iris-vpu + - audioreach + debian_revision: '0qli1' + target_workspace: qli + + # --------------------------------------------------------------------------- + # qcom-next-debug 7.2 — the same ref with the in-tree debug fragments. + # --------------------------------------------------------------------------- + - name: qcom-next-debug-trixie + suite: trixie + flavour: qcom-next-debug + git_clone: https://github.com/qualcomm-linux/kernel + branch_or_tag: qcom-next-7.2-20260826 + ref_strategy: pinned_ref + srcpkg: linux-qcom-next-debug + binpkg: linux-image-qcom-next-debug + kernel_config: + - intree:arch/arm64/configs/qcom_debug.config + - intree:kernel/configs/debug.config + dkms: + - kgsl + - camx + - iris-vpu + - audioreach + debian_revision: '0qli1~bpo13+1' + target_workspace: qli + + - name: qcom-next-debug-forky + suite: forky + flavour: qcom-next-debug + git_clone: https://github.com/qualcomm-linux/kernel + branch_or_tag: qcom-next-7.2-20260826 + ref_strategy: pinned_ref + srcpkg: linux-qcom-next-debug + binpkg: linux-image-qcom-next-debug + kernel_config: + - intree:arch/arm64/configs/qcom_debug.config + - intree:kernel/configs/debug.config + dkms: + - kgsl + - camx + - iris-vpu + - audioreach + debian_revision: '0qli1' + target_workspace: qli diff --git a/ci/scripts/resolve-matrix.py b/ci/scripts/resolve-matrix.py index 7afa7d3f..67a4d1dc 100755 --- a/ci/scripts/resolve-matrix.py +++ b/ci/scripts/resolve-matrix.py @@ -4,27 +4,39 @@ """Validate and select entries from the kernel delivery matrix. ci/build-matrix.yaml holds one entry per generated package: a single named -build, one delivery type, and one suite. Nothing here -expands or derives anything -- the matrix is already flat, and each entry -states its own debian_revision. This script validates the whole document, -selects the entries a caller asked for, and prints them. - -The document is validated in full on every invocation, not just the selected -entries, so a typo in a Release entry fails the daily build too rather than -lying in wait until someone runs a release. +build for a single suite. Nothing here expands or derives anything -- the +matrix is already flat, and each entry states its own debian_revision. This +script validates the whole document, selects the entries a caller asked for, +and prints them. + +The document has two lists of entries, both of the same shape. "builds" is +what is built nightly: every entry is built, and an entry that names a +target_workspace is also promoted into it, so the archive carries the artifact +the nightly build tested rather than a second build of the same ref. +"releases" is the other list, selected with --releases. Nothing runs its +entries on a schedule; each names the workspace it publishes into and pins the +exact ref that ships. + +Both lists are validated in full on every invocation, not just the selected +entries, so a typo in an entry nobody selected fails the run that would have +built its siblings rather than lying in wait. They are validated separately and +never mix: a name may appear in both, and means the same build in each. Usage: - ci/scripts/resolve-matrix.py --type Daily - ci/scripts/resolve-matrix.py --type Daily --build qcom-next-trixie - ci/scripts/resolve-matrix.py --type Daily --build qcom-next-trixie,qcom-next-forky - ci/scripts/resolve-matrix.py --type Release --flavour qcom-next - ci/scripts/resolve-matrix.py --type Daily --family ubuntu --allow-empty - ci/scripts/resolve-matrix.py --type Daily --build qcom-next-trixie \\ - --field debian_revision + ci/scripts/resolve-matrix.py + ci/scripts/resolve-matrix.py --build qcom-next-trixie + ci/scripts/resolve-matrix.py --build qcom-next-trixie,qcom-next-forky + ci/scripts/resolve-matrix.py --flavour qcom-next + ci/scripts/resolve-matrix.py --family ubuntu --allow-empty + ci/scripts/resolve-matrix.py --build qcom-next-trixie --field debian_revision + ci/scripts/resolve-matrix.py --releases + ci/scripts/resolve-matrix.py --releases --build qcom-next-trixie Options: - --type TYPE Delivery type to select (Daily or Release). - Required. + --releases Select from the releases list rather than from + builds. Release entries carry a + target_workspace and pin their ref, so this is + how release.yml asks what may be released. --build NAMES Select only these builds by name, comma-separated. A name identifies one build, so this is the way to ask for a specific set of them. @@ -47,9 +59,9 @@ relative to CWD). Filters combine: an entry must match every filter given. Every name in a - filter must match at least one entry of the selected type, so a typo or a - stale name fails instead of quietly narrowing the build set. That check is - per filter, so --allow-empty still rejects a name that matches nothing. + filter must match at least one entry, so a typo or a stale name fails + instead of quietly narrowing the build set. That check is per filter, so + --allow-empty still rejects a name that matches nothing. Output: Without --field, a compact JSON array of the selected entries, ready for a @@ -84,7 +96,6 @@ DEFAULT_MATRIX_FILE = "ci/build-matrix.yaml" -DELIVERY_TYPES = ("Daily", "Release") REF_STRATEGIES = ("latest_tag", "branch_tip", "pinned_ref") # Which build path a suite takes. Debian-family suites are built by Debusine; @@ -101,16 +112,9 @@ def family_for(suite): """Return the build family a suite belongs to.""" return "debian" if suite in DEBIAN_SUITES else "ubuntu" -# A delivery type constrains how its kernel ref is chosen: a Daily build tracks -# something moving, a Release build is pinned to an immutable ref. -REF_STRATEGIES_FOR_TYPE = { - "Daily": ("latest_tag", "branch_tip"), - "Release": ("pinned_ref",), -} REQUIRED_STRING_FIELDS = ( "name", - "type", "suite", "flavour", "git_clone", @@ -129,10 +133,18 @@ def family_for(suite): "debusine_parent_workspace", ) +# A release entry states its destination, and a build entry may not. This is +# the whole difference between the two lists: where a nightly build is +# published follows from why it is running, so its entry says nothing about it, +# while a release exists precisely to put one ref into one archive. +RELEASE_REQUIRED_STRING_FIELDS = ("target_workspace",) + KNOWN_FIELDS = frozenset( REQUIRED_STRING_FIELDS + OPTIONAL_STRING_FIELDS + ("kernel_config", "dkms") ) +RELEASE_KNOWN_FIELDS = KNOWN_FIELDS | frozenset(RELEASE_REQUIRED_STRING_FIELDS) + # Everything about what is built is anchored on flavour, not on the build's # name. The flavour is the kernel's own identity: it becomes the LOCALVERSION # suffix, so two flavours built from one ref produce distinct kernel releases @@ -147,22 +159,20 @@ def family_for(suite): # Fields a flavour may vary between suites but not within one. The out-of-tree # module set depends on which -dkms packages the target archive has, so -# it is a property of the flavour in a suite rather than of the flavour. A -# suite's Daily and Release must still agree: a Daily that bundles a different -# module set from the Release it precedes is not testing what will ship. +# it is a property of the flavour in a suite rather than of the flavour. SUITE_IDENTITY_FIELDS = ("dkms",) -# Fields deciding which kernel tree is built. All entries for one flavour and -# delivery type build the same source, so a stale suite cannot quietly ship a -# different kernel from its siblings. +# Fields deciding which kernel tree is built. All entries for one flavour build +# the same source, so a stale suite cannot quietly ship a different kernel from +# its siblings. REF_FIELDS = ("git_clone", "branch_or_tag", "ref_strategy", "tag_pattern") NAME_RE = re.compile(r"^[a-z0-9]+(?:[a-z0-9-]*[a-z0-9])?$") # The dispatch forms of daily.yml and release.yml take one builds field, where -# "all" means every entry of the delivery type and anything else is a list of -# names. A build actually called all would be unreachable through them, so the -# matrix may not define one. +# "all" means every entry and anything else is a list of names. A build +# actually called all would be unreachable through them, so the matrix may not +# define one. RESERVED_NAMES = ("all",) # A Debian revision: no hyphen (that would start a new revision component) and @@ -257,9 +267,15 @@ def check_dkms(entry, report): report(f"dkms entry '{module}' must be a package name stem, e.g. kgsl") -def check_entry(entry, report): +def check_entry(entry, report, release=False): """Validate one delivery entry in isolation.""" - for field in REQUIRED_STRING_FIELDS: + required = REQUIRED_STRING_FIELDS + known = KNOWN_FIELDS + if release: + required += RELEASE_REQUIRED_STRING_FIELDS + known = RELEASE_KNOWN_FIELDS + + for field in required: value = entry.get(field) if not isinstance(value, str) or not value: report(f"missing or invalid {field}") @@ -268,7 +284,10 @@ def check_entry(entry, report): if field in entry and not isinstance(entry[field], str): report(f"invalid {field}") - for field in sorted(set(entry) - KNOWN_FIELDS): + for field in sorted(set(entry) - known): + # target_workspace on a build entry is the likely version of this, and + # says something the nightly would silently ignore: a build entry can + # no more choose an archive than it can choose to run. report(f"unknown field {field}") for field in ("name", "suite", "flavour"): @@ -285,20 +304,9 @@ def check_entry(entry, report): check_kernel_config(entry, report) check_dkms(entry, report) - delivery_type = entry.get("type") - if delivery_type not in DELIVERY_TYPES: - report("type must be Daily or Release") - ref_strategy = entry.get("ref_strategy") if ref_strategy not in REF_STRATEGIES: report("ref_strategy must be " + ", ".join(REF_STRATEGIES)) - elif delivery_type in REF_STRATEGIES_FOR_TYPE: - allowed = REF_STRATEGIES_FOR_TYPE[delivery_type] - if ref_strategy not in allowed: - report( - f"{delivery_type} entries must use " - + " or ".join(f"ref_strategy={s}" for s in allowed) - ) if ref_strategy == "latest_tag": if not entry.get("tag_pattern"): @@ -306,36 +314,46 @@ def check_entry(entry, report): elif "tag_pattern" in entry: report("tag_pattern is only valid with ref_strategy=latest_tag") - if delivery_type == "Release": - if not entry.get("target_workspace"): - report("missing or invalid target_workspace") - # Promotion runs through Debusine, and only the Debian family is built - # there. An Ubuntu Release entry would build the package and then have - # no workspace to promote it from, publishing to the daily S3 path and - # reporting success without ever releasing anything. + # Promotion runs through Debusine, and only the Debian family is built + # there. An Ubuntu entry naming a target_workspace would build the package + # and then have no workspace to promote it from, publishing to the S3 path + # and reporting success without the archive ever gaining anything. + if entry.get("target_workspace"): suite = entry.get("suite") if isinstance(suite, str) and family_for(suite) != "debian": report( - f"Release entries must target a Debian suite, not {suite}; " - "promotion runs through Debusine, which builds " - + ", ".join(DEBIAN_SUITES) + f"target_workspace needs a Debian suite, not {suite}; promotion " + "runs through Debusine, which builds " + ", ".join(DEBIAN_SUITES) ) - elif "target_workspace" in entry: - report("target_workspace is only valid for Release") revision = entry.get("debian_revision") - if isinstance(revision, str) and revision: - if not REVISION_RE.match(revision): - report( - f'debian_revision "{revision}" is not a valid Debian revision ' - "(letters, digits, and . + ~ only, starting with a letter or digit)" - ) - # A trailing ~ sorts a version below the same version without it, so a - # Daily always sorts below the Release it will be superseded by. - elif delivery_type == "Daily" and not revision.endswith("~"): - report(f'debian_revision "{revision}" must end in ~ for a Daily entry') - elif delivery_type == "Release" and revision.endswith("~"): - report(f'debian_revision "{revision}" must not end in ~ for a Release entry') + if isinstance(revision, str) and revision and not REVISION_RE.match(revision): + report( + f'debian_revision "{revision}" is not a valid Debian revision ' + "(letters, digits, and . + ~ only, starting with a letter or digit)" + ) + + if not release: + return + + # A release names the exact source that ships. latest_tag would make two + # dispatches of one entry release different kernels, and branch_tip would + # make them release whatever the branch had reached by then. + if ref_strategy in REF_STRATEGIES and ref_strategy != "pinned_ref": + report( + f"ref_strategy must be pinned_ref for a release (got {ref_strategy}); " + "a release names one immutable ref" + ) + + # Promotion runs through Debusine, which is the Debian path. The Ubuntu + # path publishes to S3 and has no workspace to promote into, so an entry + # here would name a destination nothing could deliver it to. + suite = entry.get("suite") + if isinstance(suite, str) and suite and family_for(suite) != "debian": + report( + f"suite {suite} is not a Debian suite; only the Debian path can " + "promote into a workspace, so only its builds can be released" + ) def describe(entry, index): @@ -344,7 +362,7 @@ def describe(entry, index): return f"entry {index}" parts = [ str(entry[field]) - for field in ("name", "type", "suite") + for field in ("name", "suite") if isinstance(entry.get(field), str) ] return f"entry {index} ({'/'.join(parts)})" if parts else f"entry {index}" @@ -361,9 +379,8 @@ def check_consistency(builds, errors): only checked for the one thing it is used for: identifying that build uniquely. """ - by_leg = defaultdict(list) + by_name = defaultdict(list) by_flavour = defaultdict(list) - by_flavour_type = defaultdict(list) by_flavour_suite = defaultdict(list) by_package_version = defaultdict(list) flavours_by_package = defaultdict(set) @@ -372,15 +389,13 @@ def check_consistency(builds, errors): if not isinstance(entry, dict): continue flavour = entry.get("flavour") - delivery_type = entry.get("type") suite = entry.get("suite") if not isinstance(flavour, str): continue by_flavour[flavour].append(entry) - by_flavour_type[(flavour, delivery_type)].append(entry) by_flavour_suite[(flavour, suite)].append(entry) - by_leg[(delivery_type, entry.get("name"))].append(entry) + by_name[entry.get("name")].append(entry) for field in ("srcpkg", "binpkg"): if isinstance(entry.get(field), str) and entry[field]: @@ -392,17 +407,13 @@ def check_consistency(builds, errors): by_package_version[(entry["srcpkg"], entry["debian_revision"])].append(entry) # A name identifies one build: it is the Actions job name and what a - # workflow dispatch asks for. It only has to be unique within a delivery - # type, because a run only ever resolves one type, and that lets a Daily and - # its Release share a name. Two entries sharing both would give a run two + # workflow dispatch asks for. Two entries sharing one would give a run two # identically named jobs and make the dispatch filter ambiguous. - for (delivery_type, name), entries in sorted( - by_leg.items(), key=lambda item: str(item[0]) - ): + for name, entries in sorted(by_name.items(), key=lambda item: str(item[0])): if len(entries) > 1: suites = ", ".join(sorted(str(e.get("suite")) for e in entries)) errors.append( - f"name {name} is used by {len(entries)} {delivery_type} entries " + f"name {name} is used by {len(entries)} entries " f"(suites {suites}); a name identifies exactly one build" ) @@ -414,15 +425,6 @@ def check_consistency(builds, errors): f"flavour {flavour} must use one {field} across all its " "entries (got " + ", ".join(sorted(values)) + ")" ) - # A Release must be preceded by the Daily that tests it, but the - # converse does not hold: a flavour tracking a moving upstream has - # nothing immutable to pin, so it is built daily and never promoted. - types = {entry.get("type") for entry in entries} - if "Release" in types and "Daily" not in types: - errors.append( - f"flavour {flavour} has a Release entry but no Daily entry; " - "a release must be preceded by the daily build that tests it" - ) for (flavour, suite), entries in sorted( by_flavour_suite.items(), key=lambda item: str(item[0]) @@ -435,16 +437,14 @@ def check_consistency(builds, errors): f"{suite} entries (got " + ", ".join(sorted(values)) + ")" ) - for (flavour, delivery_type), entries in sorted( - by_flavour_type.items(), key=lambda item: str(item[0]) - ): + for flavour, entries in sorted(by_flavour.items()): for field in REF_FIELDS: values = {entry.get(field) for entry in entries} if len(values) > 1: rendered = ", ".join(sorted(str(v) for v in values)) errors.append( f"flavour {flavour} must build one {field} across its " - f"{delivery_type} entries (got {rendered})" + f"entries (got {rendered})" ) # Two flavours sharing a package name would overwrite each other in the @@ -465,42 +465,34 @@ def check_consistency(builds, errors): f"suites {suites}; each entry needs a revision of its own" ) - # A suite's Daily and Release differ only by the Daily's trailing ~, so the - # Daily reliably sorts below the Release that supersedes it. - for flavour, entries in sorted(by_flavour.items()): - revisions = { - (entry.get("type"), entry.get("suite")): entry.get("debian_revision") - for entry in entries - } - for (delivery_type, suite), revision in sorted( - revisions.items(), key=lambda item: str(item[0]) - ): - if delivery_type != "Daily" or not isinstance(revision, str): - continue - release = revisions.get(("Release", suite)) - if isinstance(release, str) and revision != release + "~": - errors.append( - f"flavour {flavour} suite {suite}: Daily " - f'debian_revision "{revision}" must be the Release revision ' - f'"{release}" with a trailing ~' - ) - -def validate(builds): - """Return every problem found in the matrix, as a list of messages.""" +def validate(builds, release=False): + """Return every problem found in one list of entries, as messages.""" errors = [] for index, entry in enumerate(builds): if not isinstance(entry, dict): errors.append(f"{describe(entry, index)}: delivery entries must be mappings") continue label = describe(entry, index) - check_entry(entry, lambda message, label=label: errors.append(f"{label}: {message}")) + check_entry( + entry, + lambda message, label=label: errors.append(f"{label}: {message}"), + release=release, + ) check_consistency(builds, errors) return errors -def load_matrix(path): - """Read, parse, and validate the matrix, returning its builds.""" +ROOT_KEYS = ("builds", "releases") + + +def load_matrix(path, section="builds"): + """Read and validate the whole matrix, returning one section's entries. + + Both sections are validated however few of them the caller wants, so a + broken release entry fails a nightly run too -- rather than waiting to be + found by whoever next tries to release. + """ try: with open(path, encoding="utf-8") as handle: document = yaml.safe_load(handle) @@ -514,36 +506,56 @@ def load_matrix(path): if not isinstance(document, dict): sys.exit(f"ERROR: {path}: matrix root must be a mapping with a builds key") - # builds is the whole schema. Anything else at the root is a leftover from - # an older matrix (suite_suffix_mapping, say) that would otherwise sit - # there looking authoritative while nothing read it. - unknown_root = sorted(set(document) - {"builds"}) + # builds and releases are the whole schema. Anything else at the root is a + # leftover from an older matrix (suite_suffix_mapping, say) that would + # otherwise sit there looking authoritative while nothing read it. + unknown_root = sorted(set(document) - set(ROOT_KEYS)) if unknown_root: sys.exit( f"ERROR: {path}: unknown top-level key(s) {', '.join(unknown_root)}; " - "builds is the only one" + + " and ".join(ROOT_KEYS) + " are the only ones" ) - builds = document.get("builds") - if not isinstance(builds, list): - sys.exit(f"ERROR: {path}: builds must be a list") - if not builds: - sys.exit(f"ERROR: {path}: builds must contain at least one entry") + sections = {} + for key in ROOT_KEYS: + entries = document.get(key) + # releases is optional, and a matrix with none is a matrix nothing has + # blessed yet rather than a broken one. builds is not: a matrix that + # builds nothing is a mistake in every case. + if entries is None and key != "builds": + sections[key] = [] + continue + if not isinstance(entries, list): + sys.exit(f"ERROR: {path}: {key} must be a list") + if not entries and key == "builds": + sys.exit(f"ERROR: {path}: builds must contain at least one entry") + sections[key] = entries - errors = validate(builds) + errors = [] + for key, entries in sections.items(): + errors += [ + f"{key}: {error}" for error in validate(entries, release=key == "releases") + ] if errors: sys.exit( f"ERROR: Invalid kernel delivery matrix in {path}:\n" + "\n".join(f" - {error}" for error in errors) ) + selected = sections[section] + if not selected: + sys.exit( + f"ERROR: {path}: no {section} are defined, so there is nothing to " + "select from" + ) + # Derived, never written: family follows from suite, so the matrix cannot # state one that disagrees with the suite it is built for. Attached after # validation, which rejects family as an unknown field on an entry. - for entry in builds: + for entry in selected: entry["family"] = family_for(entry["suite"]) - return builds + return selected def parse_filter(value): @@ -553,7 +565,7 @@ def parse_filter(value): return [name.strip() for name in value.split(",") if name.strip()] -def select(builds, delivery_type, filters): +def select(builds, filters): """Return the entries matching the requested filters, in matrix order. filters maps a field name to the list of values allowed for it, or to @@ -562,51 +574,47 @@ def select(builds, delivery_type, filters): return [ entry for entry in builds - if entry["type"] == delivery_type - and all( + if all( wanted is None or entry[field] in wanted for field, wanted in filters.items() ) ] -def unmatched_filters(builds, delivery_type, filters): - """Names asked for that no entry of this delivery type offers. +def unmatched_filters(builds, filters): + """Names asked for that no entry offers. family is exempt: it routes a selection to a build path rather than naming something in the matrix, argparse already restricts it to a real family, - and a type having no builds on one path is an ordinary state of the matrix - rather than a mistake in the request. + and a matrix having no builds on one path is an ordinary state of it rather + than a mistake in the request. """ missing = [] for field, wanted in filters.items(): if wanted is None or field == "family": continue - available = { - entry[field] for entry in builds if entry["type"] == delivery_type - } + available = {entry[field] for entry in builds} for name in wanted: if name not in available: missing.append( - f"no {delivery_type} entry has {field} {name} " + f"no entry has {field} {name} " f"(available: {', '.join(sorted(available))})" ) return missing -def describe_selection(delivery_type, filters): +def describe_selection(filters): """Render the active filters for an error message.""" - parts = [f"type={delivery_type}"] - parts += [ + parts = [ f"{field}={','.join(wanted)}" for field, wanted in filters.items() if wanted is not None ] - return " ".join(parts) + return " ".join(parts) if parts else "the whole matrix" def for_workflow(entry): - """Shape one entry the way build-kernel-deb.yml's inputs expect it.""" + """Shape one entry the way the build workflows' inputs expect it.""" return { **entry, "kernel_config": ",".join(entry["kernel_config"]), @@ -620,7 +628,9 @@ def main(): epilog="See the module docstring in this file for full documentation.", ) parser.add_argument( - "--type", required=True, choices=DELIVERY_TYPES, help="delivery type to select" + "--releases", + action="store_true", + help="select from the releases list rather than from builds", ) parser.add_argument( "--build", @@ -654,26 +664,28 @@ def main(): ) args = parser.parse_args() - builds = load_matrix(args.matrix_file) + builds = load_matrix( + args.matrix_file, section="releases" if args.releases else "builds" + ) filters = { "name": parse_filter(args.build), "flavour": parse_filter(args.flavour), "suite": parse_filter(args.suite), "family": parse_filter(args.family), } - what = describe_selection(args.type, filters) + what = describe_selection(filters) # Report a name that matches nothing before reporting an empty selection: - # "no Daily entry has name qcom-next" says what to fix, where - # "no entries found" leaves the reader to work out which filter was wrong. - unmatched = unmatched_filters(builds, args.type, filters) + # "no entry has name qcom-nxt" says what to fix, where "no entries found" + # leaves the reader to work out which filter was wrong. + unmatched = unmatched_filters(builds, filters) if unmatched: sys.exit( f"ERROR: Nothing to build for {what}:\n" + "\n".join(f" - {problem}" for problem in unmatched) ) - selected = select(builds, args.type, filters) + selected = select(builds, filters) if not selected: # A caller splitting one dispatch across both build paths asks each # family for the same selection, and one of them legitimately has From 13ebd7d82852172f95577cd9ac8c6ab4c60a0fd7 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 20/59] ci: drop the debug-build input Setting it broke the build. The input threaded from daily.yml through both family workflows into prepare-kernel-source, which turned it into --debug on the prepare-source.sh command line -- an option prepare-source.sh does not have, and whose argument parser rejects an unknown one outright. Anyone ticking the box on a dispatch got "Unknown option: --debug" and a failed prepare job, so the toggle has only ever been a way to break a run. Nothing is lost by removing it. A debug kernel is a flavour, not a run-time switch: qcom-next-debug builds the same ref with the in-tree debug fragments and packages it separately, so it installs alongside qcom-next and is built and published every night like anything else in the matrix. That is what the input's own help text already pointed people at. The promote job's guard loses a clause with it and keeps its meaning: a dispatch that overrides what gets built still promotes nowhere, and there are now two ways to do that rather than three. Signed-off-by: Christopher Obbard --- .../actions/prepare-kernel-source/action.yml | 7 +------ .github/workflows/build-kernel-debian.yml | 6 ------ .github/workflows/build-kernel-ubuntu.yml | 5 ----- .github/workflows/daily.yml | 7 ------- README.md | 17 ++++++++--------- 5 files changed, 9 insertions(+), 33 deletions(-) diff --git a/.github/actions/prepare-kernel-source/action.yml b/.github/actions/prepare-kernel-source/action.yml index 3811def2..5b2f45df 100644 --- a/.github/actions/prepare-kernel-source/action.yml +++ b/.github/actions/prepare-kernel-source/action.yml @@ -77,7 +77,7 @@ inputs: description: 'Out-of-tree DKMS modules to bundle, comma-separated and without the -dkms suffix' default: '' debian-revision: - description: 'Debian revision override (empty takes the Daily revision the matrix gives this build)' + description: 'Debian revision override (empty takes the revision the matrix gives this build)' default: '' localversion: description: 'LOCALVERSION override (auto-derived from the resolved ref if empty)' @@ -85,9 +85,6 @@ inputs: kver-extra: description: 'Extra package-version suffix (e.g. -ci42)' default: '' - debug-build: - description: 'Enable the debug build configuration' - default: 'false' qcom-next-pr: description: 'qcom-next PR numbers to merge, space-separated' default: '' @@ -387,7 +384,6 @@ runs: KVER_EXTRA_INPUT: ${{ inputs.kver-extra }} KERNEL_CONFIG_INPUT: ${{ inputs.kernel-config }} DKMS_INPUT: ${{ inputs.dkms }} - DEBUG_BUILD_INPUT: ${{ inputs.debug-build }} KERNEL_DIR: ${{ steps.clone.outputs.kernel_dir }} DISTRO: ${{ steps.env.outputs.distro }} DOCKER_DISTRO: ${{ steps.env.outputs.docker_distro }} @@ -426,7 +422,6 @@ runs: # Always passed, unlike the optional inputs above: an empty list is a # genuine "bundle nothing" rather than a request for some default. ARGS+=(--dkms "$DKMS_INPUT") - [[ "$DEBUG_BUILD_INPUT" == "true" ]] && ARGS+=(--debug) docker run -i --rm \ -v "$PWD:$PWD" --workdir="$PWD" \ diff --git a/.github/workflows/build-kernel-debian.yml b/.github/workflows/build-kernel-debian.yml index 46ff68da..02b1afb9 100644 --- a/.github/workflows/build-kernel-debian.yml +++ b/.github/workflows/build-kernel-debian.yml @@ -88,10 +88,6 @@ on: description: 'Extra suffix appended to the package version' type: string default: '' - debug-build: - description: 'Enable the debug build configuration' - type: boolean - default: false debusine-parent-workspace: description: 'Parent Debusine workspace for CI child workspace creation' type: string @@ -185,7 +181,6 @@ jobs: debian-revision: ${{ inputs.debian-revision }} localversion: ${{ inputs.localversion }} kver-extra: ${{ inputs.kver-extra }} - debug-build: ${{ inputs.debug-build }} qcom-next-pr: ${{ inputs.qcom-next-pr }} kernel-topics-pr: ${{ inputs.kernel-topics-pr }} @@ -413,7 +408,6 @@ jobs: needs: build if: >- ${{ inputs.target-workspace != '' - && !inputs.debug-build && inputs.qcom-next-pr == '' && inputs.kernel-topics-pr == '' }} runs-on: ubuntu-latest diff --git a/.github/workflows/build-kernel-ubuntu.yml b/.github/workflows/build-kernel-ubuntu.yml index 33a4b49f..eb972300 100644 --- a/.github/workflows/build-kernel-ubuntu.yml +++ b/.github/workflows/build-kernel-ubuntu.yml @@ -81,10 +81,6 @@ on: description: 'Extra suffix appended to the package version' type: string default: '' - debug-build: - description: 'Enable the debug build configuration' - type: boolean - default: false qcom-next-pr: description: 'qcom-next PR numbers to merge (e.g. "42 43")' type: string @@ -132,7 +128,6 @@ jobs: debian-revision: ${{ inputs.debian-revision }} localversion: ${{ inputs.localversion }} kver-extra: ${{ inputs.kver-extra }} - debug-build: ${{ inputs.debug-build }} qcom-next-pr: ${{ inputs.qcom-next-pr }} kernel-topics-pr: ${{ inputs.kernel-topics-pr }} diff --git a/.github/workflows/daily.yml b/.github/workflows/daily.yml index e9d06fc1..35c39914 100644 --- a/.github/workflows/daily.yml +++ b/.github/workflows/daily.yml @@ -41,11 +41,6 @@ on: # overrides, so they are inputs rather than entry fields. They apply to # every selected build, and a scheduled run leaves them at their # defaults. - debug-build: - description: 'Advanced: enable the debug build configuration' - required: false - default: false - type: boolean qcom-next-pr: description: 'Advanced Qualcomm-only override: qcom-next PR numbers to merge (e.g. "42 43")' required: false @@ -168,7 +163,6 @@ jobs: target-workspace: ${{ matrix.target_workspace || '' }} # Dispatch-only validation overrides. A scheduled run carries no inputs, # so each falls back to the off value and builds the matrix as written. - debug-build: ${{ inputs.debug-build || false }} qcom-next-pr: ${{ inputs.qcom-next-pr || '' }} kernel-topics-pr: ${{ inputs.kernel-topics-pr || '' }} secrets: @@ -202,6 +196,5 @@ jobs: debian-revision: ${{ matrix.debian_revision }} localversion: ${{ matrix.localversion || '' }} kver-extra: ${{ matrix.kver_extra || '' }} - debug-build: ${{ inputs.debug-build || false }} qcom-next-pr: ${{ inputs.qcom-next-pr || '' }} kernel-topics-pr: ${{ inputs.kernel-topics-pr || '' }} diff --git a/README.md b/README.md index ff5b80a1..8fdbd15c 100644 --- a/README.md +++ b/README.md @@ -164,10 +164,10 @@ promotion when it does. A run that skips this way is green: nothing was wrong, there was simply nothing new upstream. Three kinds of run deliberately promote nowhere: a PR build, which never -forwards a `target_workspace`; a dispatch setting any of the advanced -validation overrides (`debug-build`, `qcom-next-pr`, `kernel-topics-pr`), -whose kernel is by definition not the one the matrix describes; and any entry -that names no workspace. +forwards a `target_workspace`; a dispatch setting either of the advanced +validation overrides (`qcom-next-pr`, `kernel-topics-pr`), whose kernel is by +definition not the one the matrix describes; and any entry that names no +workspace. ### Release @@ -558,14 +558,13 @@ promoting: | `upstream-version` | None, required | The version to release without its Debian revision, e.g. `7.2.0~rc7+20260821`. Each selected entry promotes this plus its own `debian_revision`. | | `release-workspace` | `qli` | The Debusine workspace to promote into. | -`daily` carries three further inputs, which the matrix deliberately says -nothing about because they belong to a one-off validation run rather than to a -delivery target. They apply to every selected build, and a scheduled run leaves -them at their defaults: +`daily` carries two further inputs, which the matrix deliberately says nothing +about because they belong to a one-off validation run rather than to a delivery +target. They apply to every selected build, and a scheduled run leaves them at +their defaults: | Input | Default | Purpose | | --- | --- | --- | -| `debug-build` | `false` | Advanced debug configuration toggle. For a lasting debug kernel, use the `qcom-next-debug` flavour instead. | | `qcom-next-pr` | Empty | Advanced Qualcomm-only override: `qcom-next` PR numbers to merge before building. | | `kernel-topics-pr` | Empty | Advanced Qualcomm-only override: `kernel-topics` PR numbers to apply as patches. | From 45e4a4589fe97ef620e2647dde1c45319c26b1c6 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 21/59] ci: drop the PR-override inputs from the workflows qcom-next-pr and kernel-topics-pr were dispatch fields on daily.yml, threaded through both family workflows to reach prepare-kernel-source. They described a kernel the matrix does not: a tree with pull requests merged or patches applied on top of the resolved ref, built at the same version as the nightly that has neither. Now that a build promotes into an archive, a one-off validation run reaching a dispatch field is a way to put an unreviewed tree in front of everyone who installs from it, so the fields go and the promote guard loses the clauses that existed to catch them. The action keeps both inputs and the steps that act on them. Merging a PR before building is still a real thing to want; it is simply no longer something a dispatch form offers, and a caller that wants it has to say so deliberately. daily.yml is left with one input, builds, which is the whole of what a manual run has to decide. The promote guard is now just target-workspace: pr-build.yml is the caller that names none, so a pull request's kernel still reaches no archive. Signed-off-by: Christopher Obbard --- .github/workflows/build-kernel-debian.yml | 25 ++++------------------ .github/workflows/build-kernel-ubuntu.yml | 10 --------- .github/workflows/daily.yml | 25 +--------------------- README.md | 26 ++++++----------------- 4 files changed, 12 insertions(+), 74 deletions(-) diff --git a/.github/workflows/build-kernel-debian.yml b/.github/workflows/build-kernel-debian.yml index 02b1afb9..0e507ae7 100644 --- a/.github/workflows/build-kernel-debian.yml +++ b/.github/workflows/build-kernel-debian.yml @@ -100,14 +100,6 @@ on: reach an archive. type: string default: '' - qcom-next-pr: - description: 'qcom-next PR numbers to merge (e.g. "42 43")' - type: string - default: '' - kernel-topics-pr: - description: 'kernel-topics PR numbers to apply as patches' - type: string - default: '' promote-environment: description: > GitHub environment the promote job runs in, and so which approvals @@ -181,8 +173,6 @@ jobs: debian-revision: ${{ inputs.debian-revision }} localversion: ${{ inputs.localversion }} kver-extra: ${{ inputs.kver-extra }} - qcom-next-pr: ${{ inputs.qcom-next-pr }} - kernel-topics-pr: ${{ inputs.kernel-topics-pr }} # Only needed to ask the target workspace what it already has, so it is # checked out only when there is a target workspace to ask. @@ -389,13 +379,9 @@ jobs: # promoting a nightly afterwards -- by then there would be nothing left to # promote from. # - # The if: is the whole of the decision. A caller that named no workspace - # promotes nowhere, and a run carrying any of the validation overrides - # promotes nowhere either: those builds are deliberately not the kernel the - # matrix describes, and an archive must only ever receive that one. Guarding - # it here rather than in the caller means it holds for every caller, - # including a future one that forwards target-workspace without thinking - # about the overrides. + # The if: is the whole of the decision: a caller that named no workspace + # promotes nowhere. pr-build.yml is that caller, so a pull request's kernel + # never reaches an archive. # # Which archive, and what stands in front of it, is the caller's: # promote-environment carries the approval gate, so a release into qli can @@ -406,10 +392,7 @@ jobs: # build alone: it already needs prepare, so a run prepare skipped as # already published skips the build and this job with it. needs: build - if: >- - ${{ inputs.target-workspace != '' - && inputs.qcom-next-pr == '' - && inputs.kernel-topics-pr == '' }} + if: ${{ inputs.target-workspace != '' }} runs-on: ubuntu-latest permissions: contents: read # actions/checkout diff --git a/.github/workflows/build-kernel-ubuntu.yml b/.github/workflows/build-kernel-ubuntu.yml index eb972300..5a565458 100644 --- a/.github/workflows/build-kernel-ubuntu.yml +++ b/.github/workflows/build-kernel-ubuntu.yml @@ -81,14 +81,6 @@ on: description: 'Extra suffix appended to the package version' type: string default: '' - qcom-next-pr: - description: 'qcom-next PR numbers to merge (e.g. "42 43")' - type: string - default: '' - kernel-topics-pr: - description: 'kernel-topics PR numbers to apply as patches' - type: string - default: '' permissions: contents: read @@ -128,8 +120,6 @@ jobs: debian-revision: ${{ inputs.debian-revision }} localversion: ${{ inputs.localversion }} kver-extra: ${{ inputs.kver-extra }} - qcom-next-pr: ${{ inputs.qcom-next-pr }} - kernel-topics-pr: ${{ inputs.kernel-topics-pr }} # --------------------------------------------------------------------------- # build: build the prepared source in a suite-matched container, then publish. diff --git a/.github/workflows/daily.yml b/.github/workflows/daily.yml index 35c39914..8c110fa9 100644 --- a/.github/workflows/daily.yml +++ b/.github/workflows/daily.yml @@ -37,20 +37,6 @@ on: required: true default: 'all' type: string - # The matrix says nothing about these: they are one-off validation - # overrides, so they are inputs rather than entry fields. They apply to - # every selected build, and a scheduled run leaves them at their - # defaults. - qcom-next-pr: - description: 'Advanced Qualcomm-only override: qcom-next PR numbers to merge (e.g. "42 43")' - required: false - default: '' - type: string - kernel-topics-pr: - description: 'Advanced Qualcomm-only override: kernel-topics PR numbers to apply as patches' - required: false - default: '' - type: string permissions: contents: read @@ -156,15 +142,8 @@ jobs: kver-extra: ${{ matrix.kver_extra || '' }} debusine-parent-workspace: ${{ matrix.debusine_parent_workspace || '' }} # An entry naming a workspace is promoted into it once its build - # succeeds; one that names none is built and published to S3 only. The - # entry is forwarded as written: build-kernel-debian.yml is what declines - # to promote a run carrying the validation overrides below, so a build - # that is not the plain nightly cannot reach an archive by any caller. + # succeeds; one that names none is built and published to S3 only. target-workspace: ${{ matrix.target_workspace || '' }} - # Dispatch-only validation overrides. A scheduled run carries no inputs, - # so each falls back to the off value and builds the matrix as written. - qcom-next-pr: ${{ inputs.qcom-next-pr || '' }} - kernel-topics-pr: ${{ inputs.kernel-topics-pr || '' }} secrets: DEBUSINE_USER: ${{ secrets.DEBUSINE_USER }} DEBUSINE_TOKEN: ${{ secrets.DEBUSINE_TOKEN }} @@ -196,5 +175,3 @@ jobs: debian-revision: ${{ matrix.debian_revision }} localversion: ${{ matrix.localversion || '' }} kver-extra: ${{ matrix.kver_extra || '' }} - qcom-next-pr: ${{ inputs.qcom-next-pr || '' }} - kernel-topics-pr: ${{ inputs.kernel-topics-pr || '' }} diff --git a/README.md b/README.md index 8fdbd15c..1534e631 100644 --- a/README.md +++ b/README.md @@ -163,11 +163,11 @@ version this run would produce, and skips the build, the S3 publication and the promotion when it does. A run that skips this way is green: nothing was wrong, there was simply nothing new upstream. -Three kinds of run deliberately promote nowhere: a PR build, which never -forwards a `target_workspace`; a dispatch setting either of the advanced -validation overrides (`qcom-next-pr`, `kernel-topics-pr`), whose kernel is by -definition not the one the matrix describes; and any entry that names no -workspace. +Two kinds of run promote nowhere: a PR build, which never forwards a +`target_workspace`, and any entry that names none. Everything a `daily` +dispatch can say about a build comes from the matrix entry, so there is no way +to dispatch a build that differs from the nightly one and have it reach an +archive. ### Release @@ -558,21 +558,9 @@ promoting: | `upstream-version` | None, required | The version to release without its Debian revision, e.g. `7.2.0~rc7+20260821`. Each selected entry promotes this plus its own `debian_revision`. | | `release-workspace` | `qli` | The Debusine workspace to promote into. | -`daily` carries two further inputs, which the matrix deliberately says nothing -about because they belong to a one-off validation run rather than to a delivery -target. They apply to every selected build, and a scheduled run leaves them at -their defaults: - -| Input | Default | Purpose | -| --- | --- | --- | -| `qcom-next-pr` | Empty | Advanced Qualcomm-only override: `qcom-next` PR numbers to merge before building. | -| `kernel-topics-pr` | Empty | Advanced Qualcomm-only override: `kernel-topics` PR numbers to apply as patches. | - A `daily` dispatch publishes to S3 and, for an entry that names one, promotes -into that entry's `target_workspace` — unless it sets one of the advanced -overrides above, which suppress promotion because the kernel they build is not -the one the matrix describes. Promotion into the release workspace happens only -through `release.yml`. The build workflows themselves +into that entry's `target_workspace`. Promotion into the release workspace +happens only through `release.yml`. The build workflows themselves (`build-kernel-debian.yml`, `build-kernel-ubuntu.yml`) are `workflow_call` only and cannot be dispatched: one run of each is one matrix entry, and a reusable workflow cannot fan itself out over a list. From 7eb95d98429ee6d54a51d32e205fa86ced579db7 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 22/59] ci: let the run decide where a build is published, not the entry target_workspace was a matrix field, so an entry said both what kernel to build and where the result belongs. Those are not the same kind of fact. What to build is a property of the kernel and belongs in the matrix; where it goes is a property of the run -- the nightly publishes to staging, a pull request publishes nowhere, a release publishes to production -- and the same entry takes all three paths depending only on which workflow picked it up. Keeping the destination in the entry forced the distinction to be encoded twice. The matrix carried it per entry, and every consumer then had to work around it: pr-build.yml had to remember not to forward the field, release.yml had to filter on it to find out which entries were releasable, and the resolver had to reject it on Ubuntu suites because only Debian has a promotion path at all. None of those were saying anything about a kernel. So the field is gone and each caller states its own destination. daily.yml names the staging workspace for its Debian legs, overridable through a DEBUSINE_STAGING_WORKSPACE repository variable. pr-build.yml names nothing, as before, so a pull request's kernel is still built and tested and still reaches no archive. release.yml takes from-workspace and to-workspace, defaulting to qli-staging and qli, and selects with --family debian rather than by looking for a field -- which is the real rule, since promotion runs through Debusine and only Debian is built there. One consequence worth stating: every Debian entry the nightly builds now reaches staging, mainline and next and the topic branches included. They were held out before by naming no workspace. Their packages are separate source packages under their own names, so they coexist there, and the Production gate on release.yml is what still decides that none of them reaches qli. Signed-off-by: Christopher Obbard --- .github/workflows/build-kernel-debian.yml | 6 +- .github/workflows/daily.yml | 28 ++++---- .github/workflows/pr-build.yml | 8 +-- README.md | 83 +++++++++++------------ ci/build-matrix.yaml | 41 +++++------ ci/scripts/resolve-matrix.py | 25 ++----- 6 files changed, 85 insertions(+), 106 deletions(-) diff --git a/.github/workflows/build-kernel-debian.yml b/.github/workflows/build-kernel-debian.yml index 0e507ae7..af8f4521 100644 --- a/.github/workflows/build-kernel-debian.yml +++ b/.github/workflows/build-kernel-debian.yml @@ -95,9 +95,9 @@ on: target-workspace: description: > Debusine workspace to promote the built package into, empty to - promote nowhere. Set from the matrix entry's target_workspace by the - nightly build; left empty by pr-build.yml, whose builds must not - reach an archive. + promote nowhere. The caller decides: the nightly build names the + staging workspace, and pr-build.yml leaves it empty because a pull + request's kernel must not reach an archive. type: string default: '' promote-environment: diff --git a/.github/workflows/daily.yml b/.github/workflows/daily.yml index 8c110fa9..391d54eb 100644 --- a/.github/workflows/daily.yml +++ b/.github/workflows/daily.yml @@ -8,20 +8,21 @@ name: daily # or a comma-separated list of build names. Build names are matrix values, so # a new one needs no workflow edit. # -# This is the only workflow that builds a kernel on a schedule: an entry -# naming a target_workspace is promoted into that workspace as part of its -# build, so what the staging archive carries is always something this workflow -# built and tested. Nothing promotes afterwards -- the CI workspace a build -# runs in is named after that run and does not outlive it. release.yml builds -# the ref it releases for the same reason. +# This is the only workflow that builds a kernel on a schedule, and every +# package the staging archive carries comes from a run of it: its Debian +# entries are promoted into the staging workspace as part of the build that +# produced them. Nothing promotes afterwards -- the CI workspace a build runs +# in is named after that run and does not outlive it. release.yml builds the +# ref it releases for the same reason. # # Matrix schema: ci/build-matrix.yaml # One entry is one generated package, so ci/scripts/resolve-matrix.py passes # the selected entries through as they stand. Every field (name, suite, # flavour, srcpkg, binpkg, kernel_config, dkms, debian_revision, git_clone, -# branch_or_tag, ref_strategy, target_workspace) becomes an input of the -# build workflow for the entry's family. name labels the job; flavour is what -# reaches the built kernel's version and its published paths. +# branch_or_tag, ref_strategy) becomes an input of the build workflow for the +# entry's family. name labels the job; flavour is what reaches the built +# kernel's version and its published paths. Where the result is published is +# this workflow's to decide, not the entry's. # # To add a daily build target: add an entry to ci/build-matrix.yaml. No # workflow changes needed. @@ -141,9 +142,12 @@ jobs: localversion: ${{ matrix.localversion || '' }} kver-extra: ${{ matrix.kver_extra || '' }} debusine-parent-workspace: ${{ matrix.debusine_parent_workspace || '' }} - # An entry naming a workspace is promoted into it once its build - # succeeds; one that names none is built and published to S3 only. - target-workspace: ${{ matrix.target_workspace || '' }} + # Every Debian entry the nightly builds is promoted into the staging + # workspace once its build succeeds. The destination belongs to the run + # rather than to the entry: this is the nightly, so its results are what + # staging is for. A PR build of the same entry passes nothing here and + # promotes nowhere, and release.yml promotes onward from staging. + target-workspace: ${{ vars.DEBUSINE_STAGING_WORKSPACE || 'qli-staging' }} secrets: DEBUSINE_USER: ${{ secrets.DEBUSINE_USER }} DEBUSINE_TOKEN: ${{ secrets.DEBUSINE_TOKEN }} diff --git a/.github/workflows/pr-build.yml b/.github/workflows/pr-build.yml index 3ef53b0a..5eadb602 100644 --- a/.github/workflows/pr-build.yml +++ b/.github/workflows/pr-build.yml @@ -65,10 +65,10 @@ jobs: # fail-fast is disabled so a broken trixie leg still leaves forky and # resolute results on the PR. # - # target-workspace is deliberately not passed on. An entry carrying one is - # promoted into it by the nightly build; a PR build of the same entry only - # has to prove it builds, and must not put a pull request's kernel into an - # archive anyone installs from. + # target-workspace is deliberately not passed on. The nightly build names + # the staging workspace and its results are promoted there; a PR build of the + # same entry only has to prove it builds, and must not put a pull request's + # kernel into an archive anyone installs from. # --------------------------------------------------------------------------- build-debian: name: ${{ matrix.name }} diff --git a/README.md b/README.md index 1534e631..c354029e 100644 --- a/README.md +++ b/README.md @@ -20,13 +20,13 @@ kernel flavour owns as many entries as it has suites. ### Configured variants -| Variant | Source package | Image metapackage | Suites | Promoted to | Notes | -|---------|----------------|-------------------|--------|-------------|-------| -| `qcom-next` | `linux-qcom-next` | `linux-image-qcom-next` | trixie, forky, resolute | `qli-staging` (trixie, forky) | Standard kernel | -| `qcom-next-debug` | `linux-qcom-next-debug` | `linux-image-qcom-next-debug` | trixie, forky | `qli-staging` | Adds `arch/arm64/configs/qcom_debug.config` and `kernel/configs/debug.config` from the kernel source, via `intree:` entries | -| `qcom-arduino` | `linux-qcom-arduino` | `linux-image-qcom-arduino` | trixie, forky | none | Arduino hardware-enablement topic branch (`early/hwe/arduino` of `kernel-topics`) | -| `mainline` | `linux-mainline` | `linux-image-mainline` | trixie, forky | none | Tip of Linus's tree, tracked for early warning of upstream breakage. No DKMS modules | -| `next` | `linux-next` | `linux-image-next` | trixie, forky | none | Newest `next-YYYYMMDD` tag of linux-next. No DKMS modules | +| Variant | Source package | Image metapackage | Suites | Notes | +|---------|----------------|-------------------|--------|-------| +| `qcom-next` | `linux-qcom-next` | `linux-image-qcom-next` | trixie, forky, resolute | Standard kernel | +| `qcom-next-debug` | `linux-qcom-next-debug` | `linux-image-qcom-next-debug` | trixie, forky | Adds `arch/arm64/configs/qcom_debug.config` and `kernel/configs/debug.config` from the kernel source, via `intree:` entries | +| `qcom-arduino` | `linux-qcom-arduino` | `linux-image-qcom-arduino` | trixie, forky | Arduino hardware-enablement topic branch (`early/hwe/arduino` of `kernel-topics`) | +| `mainline` | `linux-mainline` | `linux-image-mainline` | trixie, forky | Tip of Linus's tree, tracked for early warning of upstream breakage. No DKMS modules | +| `next` | `linux-next` | `linux-image-next` | trixie, forky | Newest `next-YYYYMMDD` tag of linux-next. No DKMS modules | `derive-localversion.sh` folds the *flavour* into LOCALVERSION, so each produces a distinct kernel release (`+qcom-next--g`, @@ -35,9 +35,8 @@ image package that can be installed alongside the others. The flavour is what the kernel is; a build's `name` is only what CI calls it. See [docs/version.md](docs/version.md) for how the version strings are composed. -The last three name no `target_workspace`: they track a moving upstream for -early warning, are published to S3, and never reach an archive anyone installs -from. +The last three track a moving upstream for early warning. They are built and +promoted like the rest; nothing about an entry says where it goes. `ci/build-matrix.yaml` is the source of truth; this table is a summary. @@ -79,7 +78,6 @@ builds: - iris-vpu - audioreach debian_revision: '0qli1~bpo13+1~' - target_workspace: qli-staging ``` Entries are written out in full rather than sharing YAML anchors, so each one @@ -150,8 +148,8 @@ comes from a run of it. the configured branch directly. - Debian suites build in Debusine, then their `.deb` outputs are downloaded and uploaded to the configured S3 bucket. -- An entry naming a `target_workspace` is then promoted into it with Debusine's - `package-publish` workflow, making it installable from that archive. +- Debian entries are then promoted into the staging workspace with Debusine's + `package-publish` workflow, making them installable from that archive. - `resolute` stays on the Docker-based Ubuntu path and uploads its package outputs to the existing temporary-package S3 location. @@ -163,11 +161,11 @@ version this run would produce, and skips the build, the S3 publication and the promotion when it does. A run that skips this way is green: nothing was wrong, there was simply nothing new upstream. -Two kinds of run promote nowhere: a PR build, which never forwards a -`target_workspace`, and any entry that names none. Everything a `daily` -dispatch can say about a build comes from the matrix entry, so there is no way -to dispatch a build that differs from the nightly one and have it reach an -archive. +Where a build is published belongs to the run, not to the entry: the nightly +names the staging workspace, and `pr-build.yml` names nothing, so a pull +request's kernel is built and tested but reaches no archive. Everything a +`daily` dispatch can say about a build comes from the matrix entry, so there is +no way to dispatch a build that differs from the nightly one at all. ### Release @@ -222,9 +220,9 @@ down and reviewed before the run that ships it. #### Moving the nightly to `qli` -`qli-staging` is the nightly's destination, named by the `target_workspace` of -each nightly entry in [ci/build-matrix.yaml](ci/build-matrix.yaml). Pointing -those at `qli` would publish every night's kernel straight into the released +`qli-staging` is the nightly's destination, set by `DEBUSINE_STAGING_WORKSPACE` +and defaulted in [daily.yml](.github/workflows/daily.yml). Pointing that +variable at `qli` would publish every night's kernel straight into the released archive with nothing in between, and the approval gate that `promote-environment` provides would then have to be applied to every nightly run — stopping each of them to wait for one. The two archives, with @@ -261,14 +259,13 @@ whoever next tries to release. Each entry carries: | `debian_revision` | The Debian revision this package is built at, stated outright. Carried into every archive the package reaches, because a release promotes the built artifact rather than rebuilding it. | | `localversion`, `kver_extra` | Optional version overrides forwarded to packaging. | | `debusine_parent_workspace` | Optional parent workspace override for the variant's CI child workspaces. | -| `target_workspace` | The Debusine workspace this entry publishes into. On a `builds` entry it is optional and names where the nightly promotes it; an entry without one is built and published to S3 and reaches no archive. On a `releases` entry it is required, because putting one ref into one archive is the whole of what a release is. Debian suites only. | +| `target_workspace` | **`releases` only, and required there.** The Debusine workspace this entry publishes into. It is the one field a `builds` entry may not carry: where a nightly goes follows from why it is running, and is the calling workflow's to decide, while a release exists precisely to put one ref into one archive. | `resolve-matrix.py` rejects the matrix — before any build job starts — where an entry has an unknown field or a missing required one, a malformed variant or suite identifier, an unknown `ref_strategy`, a `tag_pattern` without -`latest_tag`, a `target_workspace` on a non-Debian suite, a `kernel_config` -fragment that escapes the kernel source root or collides with another -fragment's filename, a `dkms` entry that is not a package name stem or repeats, +`latest_tag`, a `kernel_config` fragment that escapes the kernel source root +or collides with another fragment's filename, a `dkms` entry that is not a package name stem or repeats, or a `debian_revision` that is not a valid Debian revision. A `releases` entry is held to two rules of its own: `ref_strategy` must be @@ -334,10 +331,9 @@ This document covers the CI generator. For the packaging internals: `debian/rule targets, the config fragment merge pipeline, DKMS module bundling and the produced package layout see [debian/README.md](debian/README.md). -The family branch below is taken in the caller, when the matrix is resolved, -from the entry's suite. The promotion branch is taken from the entry's own -`target_workspace`. By the time a build workflow starts, there is nothing left -to decide. +Both branches below are taken in the caller. The family follows from the +entry's suite; the promotion follows from which workflow is running. By the +time a build workflow starts, there is nothing left to decide. ```mermaid flowchart LR @@ -347,10 +343,10 @@ flowchart LR R -->|"ubuntu: resolute"| UBU["build-kernel-ubuntu.yml\nbuild-kernel.sh in Docker\nBuild binary packages"] DEB --> S3["Download .deb files\nPublish to S3"] - DEB --> TW{"target_workspace\nset?"} - TW -->|"yes, via daily.yml"| STG["Promote to qli-staging\nStaging environment"] - TW -->|"yes, via release.yml"| REL["Promote to qli\nProduction environment"] - TW -->|no| NONE["No archive"] + DEB --> TW{"Which caller"} + TW -->|"daily.yml"| STG["Promote to qli-staging\nStaging environment"] + TW -->|"release.yml"| REL["Promote to qli\nProduction environment"] + TW -->|"pr-build.yml"| NONE["No archive"] UBU --> US3["Publish .deb files to S3"] ``` @@ -373,7 +369,7 @@ flowchart TD B1["configure-matrix\nEntries, split by family"] B2["build-debian legs\nqcom-next · qcom-next-debug · qcom-arduino\nmainline · next / trixie · forky"] B5["build-ubuntu legs\nqcom-next / resolute"] - B3["configure-matrix\nEntries with a target_workspace"] + B3["configure-matrix\nDebian entries"] end subgraph build[One build workflow per leg] @@ -381,7 +377,7 @@ flowchart TD C3["build\ndebusine-build action"] C4["build\nbuild-kernel.sh in Docker"] C5["publish\nDownload .deb files, upload to S3"] - C6["promote\nlib/release into target_workspace"] + C6["promote\nlib/release into the caller's workspace"] end subgraph outputs[Outputs] @@ -445,7 +441,7 @@ flowchart LR APT --> S3["S3\npackage artifacts"] end - subgraph promote[promote job: only when target_workspace is set] + subgraph promote[promote job: only when the caller named a workspace] WS --> PROMOTE["lib/release\nStart package-publish"] PROMOTE --> STG["qli-staging (daily)\nqli (release)\nDebusine APT repository"] end @@ -539,7 +535,7 @@ Both name what to act on in one **Builds** field. | Input | Default | Purpose | | --- | --- | --- | -| `builds` | `all` | `all` selects every entry — for `release`, every entry that names a `target_workspace`. Otherwise a comma-separated list of build `name` values from `ci/build-matrix.yaml`, e.g. `qcom-next-trixie,qcom-next-debug-forky`. | +| `builds` | `all` | `all` selects every entry — for `release`, every Debian entry. Otherwise a comma-separated list of build `name` values from `ci/build-matrix.yaml`, e.g. `qcom-next-trixie,qcom-next-debug-forky`. | Everything else about a build — its suite, flavour, kernel repository and ref, package names, config fragments, DKMS modules and Debian revision — comes from @@ -556,11 +552,12 @@ promoting: | Input | Default | Purpose | | --- | --- | --- | | `upstream-version` | None, required | The version to release without its Debian revision, e.g. `7.2.0~rc7+20260821`. Each selected entry promotes this plus its own `debian_revision`. | -| `release-workspace` | `qli` | The Debusine workspace to promote into. | +| `from-workspace` | `qli-staging` | The Debusine workspace to promote out of, where the nightly build put the packages. | +| `to-workspace` | `qli` | The Debusine workspace to promote into. | -A `daily` dispatch publishes to S3 and, for an entry that names one, promotes -into that entry's `target_workspace`. Promotion into the release workspace -happens only through `release.yml`. The build workflows themselves +A `daily` dispatch publishes to S3 and promotes its Debian entries into the +staging workspace, exactly as the scheduled run does. Promotion into the +release workspace happens only through `release.yml`. The build workflows themselves (`build-kernel-debian.yml`, `build-kernel-ubuntu.yml`) are `workflow_call` only and cannot be dispatched: one run of each is one matrix entry, and a reusable workflow cannot fan itself out over a list. @@ -624,9 +621,7 @@ To add a kernel variant: `resolve-matrix.py` rejects the matrix if they drift apart. 3. Use `latest_tag` with a dated tag glob, or `branch_tip`, to track a moving upstream; `pinned_ref` freezes the variant on one ref. -4. Give the variant distinct `srcpkg` and `binpkg` values. Set - `target_workspace` on the Debian entries that should be installable from an - archive, and leave it off the ones built only for early warning. +4. Give the variant distinct `srcpkg` and `binpkg` values. 5. Give each entry a `debian_revision` that sorts where its suite belongs relative to the others and collides with no other entry building the same `srcpkg`. diff --git a/ci/build-matrix.yaml b/ci/build-matrix.yaml index 56f85e09..6a805acc 100644 --- a/ci/build-matrix.yaml +++ b/ci/build-matrix.yaml @@ -29,20 +29,20 @@ # target archive carries: the Debian suites bundle camx and iris-vpu, which # Ubuntu does not package. # -# Every entry under "builds" is built nightly. An entry that names a -# target_workspace is also promoted into it once the build succeeds, so the -# archive carries the artifact the nightly build tested rather than a rebuild -# of the same ref. mainline, next and the topic branches name no -# target_workspace: they are early-warning builds, published to S3 and nowhere -# else. +# Every entry under "builds" is built nightly, and describes a kernel rather +# than a destination: where such a build is published is decided by the +# workflow running it, because that depends on why the build is running rather +# than on what is being built. The nightly promotes its Debian entries into the +# staging workspace; a PR build of the same entry promotes nowhere. # -# "releases" is the other list, and the one place a shipped ref is written -# down. A release is a decision rather than a schedule: nothing runs these -# entries until someone dispatches release.yml, and what they name is an -# immutable ref rather than whatever the branch has reached. Each states the -# workspace it publishes into -- so the ref that ships and the archive it ships -# into are both reviewable in a pull request, rather than typed into a dispatch -# form at the moment of releasing. +# "releases" is the exception, and the one place a destination is written down. +# A release is a decision rather than a schedule: nothing runs these entries +# until someone dispatches release.yml, and what they name is an immutable ref +# rather than whatever the branch has reached. Each states the workspace it +# publishes into, because putting one ref into one archive is the whole of what +# it is -- so the ref that ships and the archive it ships into are both +# reviewable in a pull request, rather than typed into a dispatch form at the +# moment of releasing. # # The two lists are validated separately and never mix: a name may appear in # both, and means the same build in each. @@ -77,7 +77,6 @@ builds: - iris-vpu - audioreach debian_revision: '0qli1~bpo13+1~' - target_workspace: qli-staging - name: qcom-next-forky suite: forky @@ -95,7 +94,6 @@ builds: - iris-vpu - audioreach debian_revision: '0qli1~' - target_workspace: qli-staging - name: qcom-next-resolute suite: resolute @@ -134,7 +132,6 @@ builds: - iris-vpu - audioreach debian_revision: '0qli1~bpo13+1~' - target_workspace: qli-staging - name: qcom-next-debug-forky suite: forky @@ -154,13 +151,11 @@ builds: - iris-vpu - audioreach debian_revision: '0qli1~' - target_workspace: qli-staging # --------------------------------------------------------------------------- # qcom-arduino — the Arduino hardware-enablement topic branch. # - # No target_workspace: a topic branch is built for early warning, not for - # anyone to install from an archive. + # A topic branch, built for early warning of what it will take to carry. # --------------------------------------------------------------------------- - name: qcom-arduino-trixie suite: trixie @@ -195,9 +190,8 @@ builds: # --------------------------------------------------------------------------- # mainline — Linus's tree, tracked for early warning of upstream breakage. # - # No target_workspace, and no DKMS: the out-of-tree modules target the - # Qualcomm tree, and a listed module is a presence contract that would fail - # the build here. + # No DKMS: the out-of-tree modules target the Qualcomm tree, and a listed + # module is a presence contract that would fail the build here. # --------------------------------------------------------------------------- - name: mainline-trixie suite: trixie @@ -227,8 +221,7 @@ builds: # next — linux-next, the integration tree, one step ahead of mainline. # # next-YYYYMMDD tags are cut daily, so latest_tag resolves the newest of - # them the same way qcom-next-* does. No target_workspace and no DKMS, as - # above. + # them the same way qcom-next-* does. No DKMS, as above. # --------------------------------------------------------------------------- - name: next-trixie suite: trixie diff --git a/ci/scripts/resolve-matrix.py b/ci/scripts/resolve-matrix.py index 67a4d1dc..74356811 100755 --- a/ci/scripts/resolve-matrix.py +++ b/ci/scripts/resolve-matrix.py @@ -9,12 +9,12 @@ script validates the whole document, selects the entries a caller asked for, and prints them. -The document has two lists of entries, both of the same shape. "builds" is -what is built nightly: every entry is built, and an entry that names a -target_workspace is also promoted into it, so the archive carries the artifact -the nightly build tested rather than a second build of the same ref. -"releases" is the other list, selected with --releases. Nothing runs its -entries on a schedule; each names the workspace it publishes into and pins the +The document has two lists of entries, both of the same shape. "builds" is what +is built nightly, and describes a kernel rather than a destination: where such +a build is published is decided by the workflow running it, not stated here -- +the nightly promotes its Debian entries into the staging workspace, and a PR +build promotes nowhere. "releases" is the exception, selected with --releases, +and each of its entries names the workspace it publishes into and pins the exact ref that ships. Both lists are validated in full on every invocation, not just the selected @@ -127,7 +127,6 @@ def family_for(suite): OPTIONAL_STRING_FIELDS = ( "tag_pattern", - "target_workspace", "localversion", "kver_extra", "debusine_parent_workspace", @@ -314,18 +313,6 @@ def check_entry(entry, report, release=False): elif "tag_pattern" in entry: report("tag_pattern is only valid with ref_strategy=latest_tag") - # Promotion runs through Debusine, and only the Debian family is built - # there. An Ubuntu entry naming a target_workspace would build the package - # and then have no workspace to promote it from, publishing to the S3 path - # and reporting success without the archive ever gaining anything. - if entry.get("target_workspace"): - suite = entry.get("suite") - if isinstance(suite, str) and family_for(suite) != "debian": - report( - f"target_workspace needs a Debian suite, not {suite}; promotion " - "runs through Debusine, which builds " + ", ".join(DEBIAN_SUITES) - ) - revision = entry.get("debian_revision") if isinstance(revision, str) and revision and not REVISION_RE.match(revision): report( From 2e4ac71f9aee3c0c770f10a017738b0262450b67 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 23/59] ci: resolve the nightly's build-dependencies from released archives only Every Debian build read "qli qli-staging" for its Build-Depends. That was right when a release was its own build from a pinned ref reading qli alone: the daily could resolve against staging because the thing that shipped resolved against released packages and would have failed if a dependency was missing there. Releasing by promotion removes that second build, and with it the check. The artifact a release publishes is now the one the nightly produced, so whatever the nightly resolved a Build-Depends against is what the released kernel depends on. Left alone, a kernel could reach qli needing a -dkms package that only ever reached qli-staging, and nothing in the pipeline would notice until someone tried to install it. So the workspace list becomes an input and each caller states its own. The nightly reads qli, because it is the build a release promotes. pr-build.yml reads qli and qli-staging, because nothing it builds is promoted anywhere and a kernel and the module it needs should be reviewable in one pull request. This is stricter than what the nightly had, so it will fail where the previous setting quietly succeeded: if a -dkms package the matrix lists has not been released to qli, the nightly build of every entry bundling it stops until it is. That failure is the point -- it is the released kernel's dependency that cannot be satisfied -- but it lands on the nightly rather than on a release, so expect it there first. Signed-off-by: Christopher Obbard --- .github/actions/debusine-build/action.yml | 7 +++---- .github/workflows/build-kernel-debian.yml | 13 ++++++++++--- .github/workflows/daily.yml | 5 +++++ .github/workflows/pr-build.yml | 5 +++++ README.md | 6 ++++++ 5 files changed, 29 insertions(+), 7 deletions(-) diff --git a/.github/actions/debusine-build/action.yml b/.github/actions/debusine-build/action.yml index 17b3140a..86a7404e 100644 --- a/.github/actions/debusine-build/action.yml +++ b/.github/actions/debusine-build/action.yml @@ -4,7 +4,7 @@ description: > Debusine, returning the CI workspace it was built in. # Called by build-kernel-debian.yml, which decides what becomes of the result: -# every build is downloaded and uploaded to S3, and one whose entry names a +# every build is downloaded and uploaded to S3, and one whose caller named a # target workspace is also promoted into it. Holding these steps in a composite # action keeps the submission itself in one place, separate from the job that # declares the runner, container and environment it happens in. @@ -41,9 +41,8 @@ inputs: extra-build-dep-workspaces: description: > Workspaces the build resolves its build-dependencies from, space - separated. The nightly build reads qli-staging as well as qli, because a - kernel may need a -dkms package version that has reached staging and not - yet been released. + separated. Set by the calling workflow, which knows whether this build is + one a release may promote. default: 'qli' debusine-action-ref: description: 'Ref of qualcomm-linux/debusine-action to use' diff --git a/.github/workflows/build-kernel-debian.yml b/.github/workflows/build-kernel-debian.yml index af8f4521..e8b8c78c 100644 --- a/.github/workflows/build-kernel-debian.yml +++ b/.github/workflows/build-kernel-debian.yml @@ -92,6 +92,15 @@ on: description: 'Parent Debusine workspace for CI child workspace creation' type: string default: '' + extra-build-dep-workspaces: + description: > + Workspaces the build resolves its Build-Depends from, space + separated. The caller decides, because it follows from what becomes + of the result: the nightly is what a release promotes, so it reads + released archives only, and a PR build may legitimately need a + package that has reached staging and no further. + type: string + default: 'qli' target-workspace: description: > Debusine workspace to promote the built package into, empty to @@ -292,9 +301,7 @@ jobs: debusine-user: ${{ secrets.DEBUSINE_USER }} debusine-token: ${{ secrets.DEBUSINE_TOKEN }} debusine-parent-workspace: ${{ inputs.debusine-parent-workspace || vars.DEBUSINE_PARENT_WORKSPACE || 'qli-ci' }} - # A staging build: also reads qli-staging, which carries package - # versions that have not been released to qli yet. - extra-build-dep-workspaces: qli qli-staging + extra-build-dep-workspaces: ${{ inputs.extra-build-dep-workspaces }} # --------------------------------------------------------------------------- # publish: download the .deb files from Debusine and upload them to S3. diff --git a/.github/workflows/daily.yml b/.github/workflows/daily.yml index 391d54eb..554c67a9 100644 --- a/.github/workflows/daily.yml +++ b/.github/workflows/daily.yml @@ -142,6 +142,11 @@ jobs: localversion: ${{ matrix.localversion || '' }} kver-extra: ${{ matrix.kver_extra || '' }} debusine-parent-workspace: ${{ matrix.debusine_parent_workspace || '' }} + # Released archives only. This build is the one a release promotes, so + # anything it resolves a Build-Depends against is something the released + # kernel will depend on: reading qli-staging here would let a kernel + # reach qli depending on a -dkms package that has not. + extra-build-dep-workspaces: qli # Every Debian entry the nightly builds is promoted into the staging # workspace once its build succeeds. The destination belongs to the run # rather than to the entry: this is the nightly, so its results are what diff --git a/.github/workflows/pr-build.yml b/.github/workflows/pr-build.yml index 5eadb602..89f14be7 100644 --- a/.github/workflows/pr-build.yml +++ b/.github/workflows/pr-build.yml @@ -99,6 +99,11 @@ jobs: # Mark the version so PR artifacts never collide with the daily build's # in the shared Debusine workspace. kver-extra: -pr${{ github.event.number }} + # Staging as well as released: nothing here is promoted anywhere, so a + # pull request may depend on a -dkms package that has reached qli-staging + # and not yet qli -- which is how a kernel and the module it needs get + # reviewed together. + extra-build-dep-workspaces: qli qli-staging secrets: DEBUSINE_USER: ${{ secrets.DEBUSINE_USER }} DEBUSINE_TOKEN: ${{ secrets.DEBUSINE_TOKEN }} diff --git a/README.md b/README.md index c354029e..a190bd28 100644 --- a/README.md +++ b/README.md @@ -150,6 +150,12 @@ comes from a run of it. uploaded to the configured S3 bucket. - Debian entries are then promoted into the staging workspace with Debusine's `package-publish` workflow, making them installable from that archive. +- Debian builds resolve their Build-Depends against `qli` alone. The nightly is + the build a release promotes, so anything it builds against is something the + released kernel will depend on; reading `qli-staging` here would let a kernel + reach `qli` depending on a `-dkms` package that has not. A PR build resolves + against `qli qli-staging`, because nothing it produces is promoted and a + kernel and the module it needs should be reviewable together. - `resolute` stays on the Docker-based Ubuntu path and uploads its package outputs to the existing temporary-package S3 location. From 8c4772edb7f344e71420275329d1dce00898f18a Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Tue, 15 Sep 2026 05:08:54 +0100 Subject: [PATCH 24/59] ci: say why the nightly publishes to staging rather than qli The nightly's destination is a repository variable with a default, which says what it is but not why it is that and not the released archive. Someone retargeting it at qli would be changing an unattended job's destination to the archive people install from, and nothing next to the setting said so. Write it down where the value is set, and note what else would have to move with it: the promote job's approval gate lives in the GitHub environment its caller names, so pointing the nightly at qli without moving it to Production would publish every night's build into the released archive with nothing in front of it. Also correct the matrix and README notes on what reaches an archive: what is published is the artifact the build produced, never a rebuild at a tidier version, and that holds for every caller rather than being something release.yml does on its own. Signed-off-by: Christopher Obbard --- .github/workflows/daily.yml | 16 +++++++++++----- README.md | 15 ++++++++------- ci/build-matrix.yaml | 6 +++--- 3 files changed, 22 insertions(+), 15 deletions(-) diff --git a/.github/workflows/daily.yml b/.github/workflows/daily.yml index 554c67a9..5383c0ea 100644 --- a/.github/workflows/daily.yml +++ b/.github/workflows/daily.yml @@ -147,11 +147,17 @@ jobs: # kernel will depend on: reading qli-staging here would let a kernel # reach qli depending on a -dkms package that has not. extra-build-dep-workspaces: qli - # Every Debian entry the nightly builds is promoted into the staging - # workspace once its build succeeds. The destination belongs to the run - # rather than to the entry: this is the nightly, so its results are what - # staging is for. A PR build of the same entry passes nothing here and - # promotes nowhere, and release.yml promotes onward from staging. + # Every Debian entry the nightly builds is promoted into this workspace + # once its build succeeds. The destination belongs to the run rather than + # to the entry: this is the nightly, so its results are what the archive + # is for. A PR build of the same entry passes nothing here and promotes + # nowhere. + # + # qli-staging rather than qli: a nightly promotion is unattended, so it + # publishes where an unreviewed kernel belongs. Pointing this at qli -- + # by setting the repository variable -- would put every night's build + # straight into the released archive, so move the promote job to the + # Production environment first, and take its approval gate with it. target-workspace: ${{ vars.DEBUSINE_STAGING_WORKSPACE || 'qli-staging' }} secrets: DEBUSINE_USER: ${{ secrets.DEBUSINE_USER }} diff --git a/README.md b/README.md index a190bd28..8ac6b24c 100644 --- a/README.md +++ b/README.md @@ -150,12 +150,13 @@ comes from a run of it. uploaded to the configured S3 bucket. - Debian entries are then promoted into the staging workspace with Debusine's `package-publish` workflow, making them installable from that archive. -- Debian builds resolve their Build-Depends against `qli` alone. The nightly is - the build a release promotes, so anything it builds against is something the - released kernel will depend on; reading `qli-staging` here would let a kernel - reach `qli` depending on a `-dkms` package that has not. A PR build resolves - against `qli qli-staging`, because nothing it produces is promoted and a - kernel and the module it needs should be reviewable together. +- Debian builds resolve their Build-Depends against `qli` alone. The nightly + is the build whose output is published, so anything it builds against is + something the published kernel depends on; reading `qli-staging` here would + let a kernel be published depending on a `-dkms` package that has not been + released. A PR build resolves against `qli qli-staging`, because nothing it + produces is promoted and a kernel and the module it needs should be + reviewable together. - `resolute` stays on the Docker-based Ubuntu path and uploads its package outputs to the existing temporary-package S3 location. @@ -262,7 +263,7 @@ whoever next tries to release. Each entry carries: | `binpkg` | Kernel image metapackage name. | | `kernel_config` | Extra fragments applied on top of `debian/config-available/`, all of which is applied to every build, one per list element. A bare name selects `debian/config-available/.config`; an `intree:` entry names a fragment shipped by the kernel source, as a path relative to the kernel source root (e.g. `intree:arch/arm64/configs/qcom_debug.config`), so it stays versioned with the kernel it targets. Empty for variants that need nothing beyond `config-available/`; today it carries only `intree:` fragments. `resolve-matrix.py` joins it into the comma-separated `kernel-config` workflow input. | | `dkms` | Out-of-tree DKMS modules built and bundled into the image package, one per list element, each named without the `-dkms` suffix (e.g. `kgsl`). Each needs a `-dkms` package in the suite being built for, so this varies between suites. An empty list bundles nothing. A listed module is a presence contract: a build fails rather than shipping an image without it. `resolve-matrix.py` joins it into the comma-separated `dkms` workflow input, which reaches `prepare-source.sh --dkms`; see [debian/README.md](debian/README.md) for what the packaging does with it. | -| `debian_revision` | The Debian revision this package is built at, stated outright. Carried into every archive the package reaches, because a release promotes the built artifact rather than rebuilding it. | +| `debian_revision` | The Debian revision this package is built at, stated outright. Carried into the archive as built, because publishing promotes the artifact rather than rebuilding it. | | `localversion`, `kver_extra` | Optional version overrides forwarded to packaging. | | `debusine_parent_workspace` | Optional parent workspace override for the variant's CI child workspaces. | | `target_workspace` | **`releases` only, and required there.** The Debusine workspace this entry publishes into. It is the one field a `builds` entry may not carry: where a nightly goes follows from why it is running, and is the calling workflow's to decide, while a release exists precisely to put one ref into one archive. | diff --git a/ci/build-matrix.yaml b/ci/build-matrix.yaml index 6a805acc..f9fa34e3 100644 --- a/ci/build-matrix.yaml +++ b/ci/build-matrix.yaml @@ -51,9 +51,9 @@ # Debusine, which does not build the Ubuntu family. # # debian_revision is stated outright, not derived. The trailing "~" marks a -# version as unreleased, and it is carried into the archive as built: what -# release.yml promotes is the tested artifact, not a rebuild at a tidier -# version. The rest of the revision places the suite where it belongs relative +# version as unreleased, and it is carried into the archive as built: what is +# published is the tested artifact, never a rebuild at a tidier version. The +# rest of the revision places the suite where it belongs relative # to the other suites (see the ordering discussion in README.md). No two # entries may produce the same srcpkg at the same debian_revision. From 78915603931af82533b716a6604a1c4711983b83 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 25/59] ci: state pr-build's empty target-workspace outright The promote job in build-kernel-debian.yml runs when target-workspace is not empty, and pr-build.yml kept a pull request's kernel out of the archives by not passing the input at all -- relying on its default being empty, with the reason recorded in a comment two dozen lines above the block it applied to. That is a lot of weight for an absence to carry. A PR is unreviewed by definition and its packages are versioned -pr, so promoting one would put a kernel nobody approved into an archive people install from, at a version no later build supersedes. Nothing in the with: block said so; a change to the input's default, or a copy of the block into a workflow that should promote, would have flipped it silently. So the value is written where it takes effect, with the reason next to it. Nothing changes at runtime: the input already defaulted to empty and still does. Signed-off-by: Christopher Obbard --- .github/workflows/pr-build.yml | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/.github/workflows/pr-build.yml b/.github/workflows/pr-build.yml index 89f14be7..8da28d8f 100644 --- a/.github/workflows/pr-build.yml +++ b/.github/workflows/pr-build.yml @@ -64,11 +64,6 @@ jobs: # # fail-fast is disabled so a broken trixie leg still leaves forky and # resolute results on the PR. - # - # target-workspace is deliberately not passed on. The nightly build names - # the staging workspace and its results are promoted there; a PR build of the - # same entry only has to prove it builds, and must not put a pull request's - # kernel into an archive anyone installs from. # --------------------------------------------------------------------------- build-debian: name: ${{ matrix.name }} @@ -104,6 +99,19 @@ jobs: # and not yet qli -- which is how a kernel and the module it needs get # reviewed together. extra-build-dep-workspaces: qli qli-staging + # Empty, and stated rather than left to the input's default: this is the + # one value keeping a pull request's kernel out of the archives. An empty + # target-workspace is what build-kernel-debian.yml's promote job tests + # for, so a PR is built, tested and published to S3 and goes no further. + # + # A PR is unreviewed by definition, and its packages are versioned + # -pr rather than as the nightly's, so promoting one would put a + # kernel nobody has approved into an archive people install from, under a + # version no release will ever supersede. Left implicit, a future edit to + # the default -- or a copy of this block into a workflow that should + # promote -- would flip that silently; spelled out here, changing it + # means saying so. + target-workspace: '' secrets: DEBUSINE_USER: ${{ secrets.DEBUSINE_USER }} DEBUSINE_TOKEN: ${{ secrets.DEBUSINE_TOKEN }} From cc0859fbd458265a894bb6fbbaabf47055d5c832 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 26/59] packaging: drop the trailing ~ from every Debian revision The trailing ~ existed to sort a daily build below the release built from the same ref: two builds of one kernel needed two versions, and ~ is what made the daily the lesser of them. resolve-matrix.py enforced the pairing, requiring a Daily revision to be its Release revision plus a ~. There is no second build any more. A kernel is built once, and the artifact that build produced is what any archive holds, so nothing exists for a ~ to sort against. All it does now is make every published package look permanently provisional -- a version that will never be superseded by the non-~ one it implies is coming. So trixie builds at 0qli1~bpo13+1, forky at 0qli1, resolute at 0qli1~26.04.1. The remaining ~ in the trixie and resolute revisions is the backports convention and stays: it is what sorts a trixie backport below a forky build of the same kernel, and the cross-suite order the README documents is unchanged -- resolute < trixie < forky, verified with dpkg --compare-versions. Upgrades are unaffected. Every new version sorts above the ~-suffixed one it replaces, so a machine tracking qli-staging moves onto the next nightly by itself. The first run after this builds and publishes rather than skipping, because the version it produces is genuinely one the archive has not seen. Signed-off-by: Christopher Obbard --- .github/workflows/build-kernel-debian.yml | 2 +- .github/workflows/build-kernel-ubuntu.yml | 2 +- README.md | 21 +++++++------ ci/build-matrix.yaml | 36 ++++++++++++----------- docs/version.md | 22 +++++++------- 5 files changed, 43 insertions(+), 40 deletions(-) diff --git a/.github/workflows/build-kernel-debian.yml b/.github/workflows/build-kernel-debian.yml index e8b8c78c..2eb4b0f7 100644 --- a/.github/workflows/build-kernel-debian.yml +++ b/.github/workflows/build-kernel-debian.yml @@ -79,7 +79,7 @@ on: debian-revision: description: 'Debian revision component of the package version' type: string - default: '0qli~' + default: '0qli1' localversion: description: 'Override LOCALVERSION suffix (auto-derived from the resolved ref if empty)' type: string diff --git a/.github/workflows/build-kernel-ubuntu.yml b/.github/workflows/build-kernel-ubuntu.yml index 5a565458..c0c36423 100644 --- a/.github/workflows/build-kernel-ubuntu.yml +++ b/.github/workflows/build-kernel-ubuntu.yml @@ -72,7 +72,7 @@ on: debian-revision: description: 'Debian revision component of the package version' type: string - default: '0qli~' + default: '0qli1' localversion: description: 'Override LOCALVERSION suffix (auto-derived from the resolved ref if empty)' type: string diff --git a/README.md b/README.md index 8ac6b24c..f3126f76 100644 --- a/README.md +++ b/README.md @@ -77,7 +77,7 @@ builds: - camx - iris-vpu - audioreach - debian_revision: '0qli1~bpo13+1~' + debian_revision: '0qli1~bpo13+1' ``` Entries are written out in full rather than sharing YAML anchors, so each one @@ -89,16 +89,15 @@ Each entry states its `debian_revision` outright. The configured values are: | Suite | Revision | | --- | --- | -| Trixie | `0qli~bpo13+1~` | -| Forky | `0qli~` | -| Resolute | `0qli~26.04.1~` | - -The trailing `~` marks the version as one nobody has blessed yet, and it stays -on the version through release. That is a consequence of releasing by -promotion: what reaches `qli` is the artifact `qli-staging` holds, byte for -byte, so its version is the version it was built with. A release that dropped -the `~` would have to be a different build, which is exactly what this design -removes. +| Trixie | `0qli1~bpo13+1` | +| Forky | `0qli1` | +| Resolute | `0qli1~26.04.1` | + +None of them carries a trailing `~`. That marker existed to sort a daily build +below the release rebuilt from the same ref, and there is no such rebuild: the +version is decided once, when the package is built, and the artifact any +archive holds is that one. With nothing for it to sort against, a trailing `~` +would only make every published kernel look permanently provisional. Ordering across *different* suites depends entirely on the configured revisions: with the values above, Resolute < Trixie < Forky, matching a diff --git a/ci/build-matrix.yaml b/ci/build-matrix.yaml index f9fa34e3..340f0df9 100644 --- a/ci/build-matrix.yaml +++ b/ci/build-matrix.yaml @@ -50,12 +50,14 @@ # Only a Debian suite may name a target_workspace. Promotion runs through # Debusine, which does not build the Ubuntu family. # -# debian_revision is stated outright, not derived. The trailing "~" marks a -# version as unreleased, and it is carried into the archive as built: what is -# published is the tested artifact, never a rebuild at a tidier version. The -# rest of the revision places the suite where it belongs relative -# to the other suites (see the ordering discussion in README.md). No two -# entries may produce the same srcpkg at the same debian_revision. +# debian_revision is stated outright, not derived. It carries no marker for +# where a build is in its life: the version is decided once, when the package is +# built, and the same artifact is what any archive later holds, so there is no +# second version for it to sort against. What the revision does say is where +# the suite belongs relative to the other suites -- "~bpo13+1" puts a trixie +# backport below a forky build of the same kernel (see the ordering discussion +# in README.md). No two entries may produce the same srcpkg at the same +# debian_revision. builds: # --------------------------------------------------------------------------- @@ -76,7 +78,7 @@ builds: - camx - iris-vpu - audioreach - debian_revision: '0qli1~bpo13+1~' + debian_revision: '0qli1~bpo13+1' - name: qcom-next-forky suite: forky @@ -93,7 +95,7 @@ builds: - camx - iris-vpu - audioreach - debian_revision: '0qli1~' + debian_revision: '0qli1' - name: qcom-next-resolute suite: resolute @@ -108,7 +110,7 @@ builds: # kgsl alone: camx-dkms and iris-vpu-dkms are not packaged for Ubuntu. dkms: - kgsl - debian_revision: '0qli1~26.04.1~' + debian_revision: '0qli1~26.04.1' # --------------------------------------------------------------------------- # qcom-next-debug — the same kernel ref built with the in-tree debug @@ -131,7 +133,7 @@ builds: - camx - iris-vpu - audioreach - debian_revision: '0qli1~bpo13+1~' + debian_revision: '0qli1~bpo13+1' - name: qcom-next-debug-forky suite: forky @@ -150,7 +152,7 @@ builds: - camx - iris-vpu - audioreach - debian_revision: '0qli1~' + debian_revision: '0qli1' # --------------------------------------------------------------------------- # qcom-arduino — the Arduino hardware-enablement topic branch. @@ -170,7 +172,7 @@ builds: - kgsl - camx - iris-vpu - debian_revision: '0qli~bpo13+1~' + debian_revision: '0qli1~bpo13+1' - name: qcom-arduino-forky suite: forky @@ -185,7 +187,7 @@ builds: - kgsl - camx - iris-vpu - debian_revision: '0qli~' + debian_revision: '0qli1' # --------------------------------------------------------------------------- # mainline — Linus's tree, tracked for early warning of upstream breakage. @@ -203,7 +205,7 @@ builds: binpkg: linux-image-mainline kernel_config: [] dkms: [] - debian_revision: '0qli~bpo13+1~' + debian_revision: '0qli1~bpo13+1' - name: mainline-forky suite: forky @@ -215,7 +217,7 @@ builds: binpkg: linux-image-mainline kernel_config: [] dkms: [] - debian_revision: '0qli~' + debian_revision: '0qli1' # --------------------------------------------------------------------------- # next — linux-next, the integration tree, one step ahead of mainline. @@ -234,7 +236,7 @@ builds: binpkg: linux-image-next kernel_config: [] dkms: [] - debian_revision: '0qli~bpo13+1~' + debian_revision: '0qli1~bpo13+1' - name: next-forky suite: forky @@ -247,7 +249,7 @@ builds: binpkg: linux-image-next kernel_config: [] dkms: [] - debian_revision: '0qli~' + debian_revision: '0qli1' # Releases. Dispatched from release.yml and never scheduled, so nothing here # runs until someone asks for it by name. diff --git a/docs/version.md b/docs/version.md index f396079d..d27aa782 100644 --- a/docs/version.md +++ b/docs/version.md @@ -99,11 +99,17 @@ The cost of `~` is one misleading reading: the version sorts below the same snapshot without a SHA, as though it preceded it. Nothing occupies that slot, because every snapshot build carries a SHA. -The revision (`0qli1~bpo13+1`) is derived separately, from the version stub, the -suite, and whether the build is a Daily or a Release. See the matrix -documentation in the top-level [README](../README.md#matrix-model); the trailing -digit on the stub is the packaging revision, bumped when the packaging changes -but the kernel snapshot does not. +The revision (`0qli1~bpo13+1`) is not derived. Each matrix entry states its own +outright, and it says two things: where the suite belongs relative to the other +suites, and which packaging built it. The `~bpo13+1` is the backports +convention, sorting a trixie build below a forky build of the same kernel; the +trailing digit on the `0qli` stub is the packaging revision, bumped when the +packaging changes but the kernel snapshot does not. See the matrix +documentation in the top-level [README](../README.md#matrix-model). + +Nothing in the revision marks how far a build has got. A kernel is built once +and the artifact that build produced is what any archive holds, so there is no +second version for a marker to sort against. ## Ordering @@ -112,15 +118,11 @@ The full chain for one suite, in the order dpkg sorts it: ```text 7.2.0~rc7+git20260820.1~g011a82096bee-0qli1~bpo13+1 first tag of the 20th 7.2.0~rc7+git20260820.2~g3f2f3ca1a81e-0qli1~bpo13+1 respin, same day -7.2.0~rc7+git20260821~gabcdef123456-0qli1~bpo13+1~ Daily, next snapshot -7.2.0~rc7+git20260821~gabcdef123456-0qli1~bpo13+1 Release of the same +7.2.0~rc7+git20260821~gabcdef123456-0qli1~bpo13+1 next snapshot 7.2.0~rc7+git20260821~gabcdef123456-0qli2~bpo13+1 packaging rebuild 7.2.0+git20260902~g123456789abc-0qli1~bpo13+1 7.2 final ``` -A Daily sorts below the Release of the same snapshot because its revision ends -in a trailing `~`. - ## Moved tags If an upstream tag is re-cut against a different commit, the SHA changes and so From ac65aad5ec3ca8d479ea7fe9acedd1a9620ddb05 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 27/59] ci: say what kver-extra marks in pr-build pr-build.yml claimed that kver-extra marks the version so PR artifacts never collide with the daily build's. It does not: debian/rules appends KVER_EXTRA to KVER and derives the package version separately, so a PR build shares the nightly's source version and differs only in uname -r and the versioned linux-image name. That is enough where it is -- a PR build has its own throwaway workspace and promotes nowhere -- but it is not a version stamp, and the comment should not invite anything to start relying on it as one. Signed-off-by: Christopher Obbard --- .github/workflows/pr-build.yml | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/.github/workflows/pr-build.yml b/.github/workflows/pr-build.yml index 8da28d8f..186e747a 100644 --- a/.github/workflows/pr-build.yml +++ b/.github/workflows/pr-build.yml @@ -91,8 +91,13 @@ jobs: dkms: ${{ matrix.dkms }} debian-revision: ${{ matrix.debian_revision }} localversion: ${{ matrix.localversion || '' }} - # Mark the version so PR artifacts never collide with the daily build's - # in the shared Debusine workspace. + # Marks the kernel release, not the package version: debian/rules + # appends KVER_EXTRA to KVER and then derives the version separately, so + # this changes uname -r and the versioned linux-image name while the + # source version stays the one the nightly would build. That is enough + # here -- a PR build has its own throwaway workspace and promotes + # nowhere -- but it is not a version stamp, and nothing should start + # relying on it as one. kver-extra: -pr${{ github.event.number }} # Staging as well as released: nothing here is promoted anywhere, so a # pull request may depend on a -dkms package that has reached qli-staging From 7a3d60eb56f692a117759eac7a77e81f50cb8d42 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 28/59] ci: drop the PR-patch inputs nothing passes prepare-kernel-source took qcom-next-pr and kernel-topics-pr and merged or applied them after the clone, but no workflow passes either: the build workflows have no input to carry them, so the two steps could never run. They could not have been wired up as they stood, either. The commit SHA and date are read from HEAD before the patches would be applied, so a patched build would name a commit it did not build. And a merge commit made at build time has no stable SHA, so the version could never be reproduced from it. An orig tarball keyed on the commit, which the source package is about to be, rules such builds out for good. Signed-off-by: Christopher Obbard --- .../actions/prepare-kernel-source/action.yml | 59 +------------------ 1 file changed, 2 insertions(+), 57 deletions(-) diff --git a/.github/actions/prepare-kernel-source/action.yml b/.github/actions/prepare-kernel-source/action.yml index 5b2f45df..299f0115 100644 --- a/.github/actions/prepare-kernel-source/action.yml +++ b/.github/actions/prepare-kernel-source/action.yml @@ -1,7 +1,7 @@ name: Prepare kernel source description: > - Resolve the kernel ref, clone it, apply any PR patches, inject debian/ and - generate the source package, then upload the prepared tree as an artifact. + Resolve the kernel ref, clone it, inject debian/ and generate the source + package, then upload the prepared tree as an artifact. # Every build takes this path, whichever family it belongs to, so it lives in a # composite action rather than a reusable workflow: the two family workflows @@ -85,12 +85,6 @@ inputs: kver-extra: description: 'Extra package-version suffix (e.g. -ci42)' default: '' - qcom-next-pr: - description: 'qcom-next PR numbers to merge, space-separated' - default: '' - kernel-topics-pr: - description: 'kernel-topics PR numbers to apply as patches, space-separated' - default: '' runs: using: composite @@ -327,55 +321,6 @@ runs: fi echo "debian_revision=$DEBIAN_REVISION" >> "$GITHUB_OUTPUT" - - name: Merge qcom-next PR patches - if: ${{ inputs.qcom-next-pr != '' }} - shell: bash - env: - QCOM_NEXT_PR: ${{ inputs.qcom-next-pr }} - KERNEL_DIR: ${{ steps.clone.outputs.kernel_dir }} - run: | - cd "$KERNEL_DIR" - echo "Merging qcom-next PR(s): $QCOM_NEXT_PR" - for pr in $QCOM_NEXT_PR; do - [[ "$pr" =~ ^[0-9]+$ ]] || { echo "::error::invalid qcom-next PR number: $pr"; exit 1; } - echo "::group::Merging qcom-next PR #$pr" - git fetch --no-tags origin "pull/$pr/head:pr-$pr" - if ! git merge --no-ff --no-commit "pr-$pr"; then - echo "ERROR: Merge conflict while merging PR #$pr. Aborting." - git merge --abort || true - exit 1 - fi - if ! git diff --cached --quiet; then - git commit -m "Merged qcom-next PR #$pr" - echo "PR #$pr merged successfully." - else - echo "PR #$pr already present; nothing to merge." - fi - echo "::endgroup::" - done - - - name: Apply kernel-topics PR patches - if: ${{ inputs.kernel-topics-pr != '' }} - shell: bash - env: - KERNEL_TOPICS_PR: ${{ inputs.kernel-topics-pr }} - KERNEL_DIR: ${{ steps.clone.outputs.kernel_dir }} - run: | - cd "$KERNEL_DIR" - echo "Applying kernel-topics PR patch(es): $KERNEL_TOPICS_PR" - for pr in $KERNEL_TOPICS_PR; do - [[ "$pr" =~ ^[0-9]+$ ]] || { echo "::error::invalid kernel-topics PR number: $pr"; exit 1; } - echo "::group::Applying kernel-topics PR #$pr" - wget -q "https://github.com/qualcomm-linux/kernel-topics/pull/$pr.patch" -O "$pr.patch" - if ! git am "$pr.patch"; then - echo "ERROR: Patch application failed for PR #$pr. Aborting." - git am --abort || true - exit 1 - fi - echo "PR #$pr applied successfully." - echo "::endgroup::" - done - - name: Prepare source shell: bash env: From 5e8c54f52891151c294c0843d1321804736e8e8e Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 29/59] ci: require debian-revision in prepare and drop the matrix lookup The action looked the revision up in ci/build-matrix.yaml when its input was empty, for a caller with no matrix entry in hand. No such caller exists: the build workflows are workflow_call only, are called with a matrix entry, and give the input a default of their own, so the lookup could not be reached. The revision is now a required input, and a caller that forgets it fails at the action boundary rather than building at a revision nobody chose. resolve-matrix.py keeps --field, which is still a useful way to ask the matrix a question from a shell. Signed-off-by: Christopher Obbard --- .../actions/prepare-kernel-source/action.yml | 36 ++----------------- README.md | 8 ++--- 2 files changed, 6 insertions(+), 38 deletions(-) diff --git a/.github/actions/prepare-kernel-source/action.yml b/.github/actions/prepare-kernel-source/action.yml index 299f0115..9e2903ae 100644 --- a/.github/actions/prepare-kernel-source/action.yml +++ b/.github/actions/prepare-kernel-source/action.yml @@ -77,8 +77,8 @@ inputs: description: 'Out-of-tree DKMS modules to bundle, comma-separated and without the -dkms suffix' default: '' debian-revision: - description: 'Debian revision override (empty takes the revision the matrix gives this build)' - default: '' + description: 'Debian revision component of the package version, from the matrix entry' + required: true localversion: description: 'LOCALVERSION override (auto-derived from the resolved ref if empty)' default: '' @@ -291,36 +291,6 @@ runs: done >> "$GITHUB_OUTPUT" fi - - name: Resolve Debian revision - # The build workflows pass the matrix entry's own debian-revision - # straight through, so this lookup is the fallback for a caller that has - # no entry in hand and leaves the input empty: it looks up the entry for - # the build name and suite it was given, and builds at the revision the - # nightly build uses. - id: revision - shell: bash - env: - DEBIAN_REVISION_INPUT: ${{ inputs.debian-revision }} - BUILD_INPUT: ${{ inputs.build }} - DISTRO: ${{ steps.env.outputs.distro }} - run: | - set -euo pipefail - if [[ -n "$DEBIAN_REVISION_INPUT" ]]; then - DEBIAN_REVISION="$DEBIAN_REVISION_INPUT" - echo "Using supplied Debian revision: $DEBIAN_REVISION" - else - DEBIAN_REVISION=$(ci/scripts/resolve-matrix.py \ - --build "$BUILD_INPUT" \ - --suite "$DISTRO" \ - --field debian_revision) || { - echo "::error::No matrix entry named $BUILD_INPUT for suite $DISTRO." - echo "::error::Add one to ci/build-matrix.yaml, or set the debian-revision input to build this combination ad hoc." - exit 1 - } - echo "Matrix Debian revision for $BUILD_INPUT/$DISTRO: $DEBIAN_REVISION" - fi - echo "debian_revision=$DEBIAN_REVISION" >> "$GITHUB_OUTPUT" - - name: Prepare source shell: bash env: @@ -334,7 +304,7 @@ runs: DOCKER_DISTRO: ${{ steps.env.outputs.docker_distro }} LOCALVERSION: ${{ steps.localversion.outputs.localversion }} SNAPSHOT: ${{ steps.localversion.outputs.snapshot }} - DEBIAN_REVISION: ${{ steps.revision.outputs.debian_revision }} + DEBIAN_REVISION: ${{ inputs.debian-revision }} KERNEL_URL: ${{ inputs.kernel-url }} KERNEL_REF: ${{ steps.ref.outputs.kernel_ref }} KERNEL_SHA: ${{ steps.clone.outputs.kernel_sha }} diff --git a/README.md b/README.md index f3126f76..ab2dcf8c 100644 --- a/README.md +++ b/README.md @@ -295,11 +295,9 @@ face value: - No `srcpkg` or `binpkg` is shared between flavours, and no two entries build the same `srcpkg` at the same `debian_revision`. -The build workflows pass their entry's own `debian_revision` through. A caller -with no entry in hand can leave the `debian-revision` input empty, and the -`prepare-kernel-source` action looks up the entry for the build name and suite -it was given (`resolve-matrix.py --field debian_revision`) and builds at the -revision the nightly build uses. +The build workflows pass their entry's own `debian_revision` through, and the +`prepare-kernel-source` action requires it: there is no caller without an +entry in hand, so there is no fallback to look one up. Each entry has a distinct prepared-source artifact, Debusine child workspace, and S3 path keyed by `flavour + suite`. This prevents two flavours that both From ad769a7951b8fce306f79b8469ac516e39308696 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 30/59] ci: say why the family guards exist and why a skipped check reads oddly The if: guards on the build jobs look removable, and are not: an empty matrix is a workflow error rather than an empty job list, so a family the run selects nothing on has to be skipped rather than started with nothing to do. Skipping one has a visible cost that looks like a bug. GitHub never expands a job it skips, so it cannot evaluate name: ${{ matrix.name }} and shows the expression's source text -- a check literally called "matrix.name". A || fallback does not help, printing its own source text just the same, and the only name that would read well is one never mentioning matrix, which would mean renaming every check in every workflow to tidy up one that appears when a family is empty. Both facts are written down so the next reader reaches that conclusion without rediscovering it. Signed-off-by: Christopher Obbard --- .github/workflows/daily.yml | 12 +++++++++--- .github/workflows/pr-build.yml | 7 ++++--- 2 files changed, 13 insertions(+), 6 deletions(-) diff --git a/.github/workflows/daily.yml b/.github/workflows/daily.yml index 5383c0ea..6062a624 100644 --- a/.github/workflows/daily.yml +++ b/.github/workflows/daily.yml @@ -110,9 +110,15 @@ jobs: # called. fail-fast is disabled so a failed trixie build does not cancel the # forky build. # - # The if: guards only matter for a narrowed dispatch that selects nothing on - # one path; an empty matrix is an error rather than an empty job list. The - # scheduled full-matrix run has entries on both. + # The if: guards catch a family the run selects nothing on -- a narrowed + # dispatch, or a matrix with no entries for it. They are not optional: an + # empty matrix is a workflow error rather than an empty job list. + # + # A skipped family shows up as a check literally named "matrix.name": GitHub + # never expands a job it skips, so it prints the expression's source text, + # and a || fallback prints just the same. Only a name that never mentions + # matrix would read well, which would mean renaming every check in every + # workflow to tidy up one that appears when a family is empty. So it stands. # --------------------------------------------------------------------------- build-debian: name: ${{ matrix.name }} diff --git a/.github/workflows/pr-build.yml b/.github/workflows/pr-build.yml index 186e747a..8f08cee2 100644 --- a/.github/workflows/pr-build.yml +++ b/.github/workflows/pr-build.yml @@ -58,9 +58,10 @@ jobs: # build-debian / build-ubuntu: one parallel job per matrix entry, mirroring # daily.yml. # - # A PR gets one check per build leg and nothing else: neither workflow has a - # branch to skip, and a PR builds the whole matrix, so both families have - # entries and neither guard fires. + # A PR builds the whole matrix, so it gets one check per build leg and + # nothing else -- unless the matrix has no entries for one family, which + # fires that family's guard and leaves a check named "matrix.name". + # daily.yml documents why the guard is needed and the name left alone. # # fail-fast is disabled so a broken trixie leg still leaves forky and # resolute results on the PR. From 1664b5bc890521ac2bab1e17bf3d7bfd500d805d Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 31/59] ci: pass the changelog date into prepare Split out of "packaging: date debian/changelog from the kernel commit": the packaging half of that change landed early, alongside the reproducible source package work, before this action existed to call it. The action reads the date on the runner and passes it in, as it does the SHA and the ref: prepare-source.sh runs in a container as root and git will not read a checkout owned by someone else. Signed-off-by: Christopher Obbard --- .github/actions/prepare-kernel-source/action.yml | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/.github/actions/prepare-kernel-source/action.yml b/.github/actions/prepare-kernel-source/action.yml index 9e2903ae..55adb59a 100644 --- a/.github/actions/prepare-kernel-source/action.yml +++ b/.github/actions/prepare-kernel-source/action.yml @@ -245,6 +245,13 @@ runs: echo "kernel_date=$KERNEL_DATE" >> "$GITHUB_OUTPUT" echo "Kernel HEAD commit date (UTC): $KERNEL_DATE" + # The same instant in RFC 2822 form, for the changelog entry. Read + # here, on the host that owns the clone, because prepare-source.sh + # runs in a container as root and git will not read a checkout owned + # by someone else. + CHANGELOG_DATE=$(git -C "$KERNEL_DIR" log -1 --format=%cD) + echo "changelog_date=$CHANGELOG_DATE" >> "$GITHUB_OUTPUT" + - name: Derive version fields id: localversion shell: bash @@ -308,6 +315,7 @@ runs: KERNEL_URL: ${{ inputs.kernel-url }} KERNEL_REF: ${{ steps.ref.outputs.kernel_ref }} KERNEL_SHA: ${{ steps.clone.outputs.kernel_sha }} + CHANGELOG_DATE: ${{ steps.clone.outputs.changelog_date }} run: | # No -x here: keep xtrace off for this step as it assembles the # prepare-source.sh argument list which may include sensitive paths. @@ -319,6 +327,7 @@ runs: --srcpkg "$SRCPKG_INPUT" --binpkg "$BINPKG_INPUT" --debian-revision "$DEBIAN_REVISION" + --changelog-date "$CHANGELOG_DATE" ) # The version fields all come from derive-localversion.sh, which From 7da0aa6db7d8dd6e516fd41f0d5253b129962b23 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:23 +0100 Subject: [PATCH 32/59] ci: require the inputs that describe the build Every input naming what a build is -- the entry, its suite and flavour, the kernel repository and ref, the package names and the Debian revision -- had a default spelling out the qcom-next entry. No caller relied on one: daily.yml, pr-build.yml and release.yml pass all of them from the matrix. What the defaults did was let a caller that dropped one build the default entry's kernel under another entry's name without anything noticing. They are now required, so an omission is a workflow error at parse time. The inputs that keep a default are the ones for which empty is a value: tag-pattern outside latest_tag, an empty fragment or module list, and the overrides and destinations the caller chooses rather than the entry. Signed-off-by: Christopher Obbard --- .github/workflows/build-kernel-debian.yml | 18 +++++++++--------- .github/workflows/build-kernel-ubuntu.yml | 18 +++++++++--------- 2 files changed, 18 insertions(+), 18 deletions(-) diff --git a/.github/workflows/build-kernel-debian.yml b/.github/workflows/build-kernel-debian.yml index 2eb4b0f7..bcc0794c 100644 --- a/.github/workflows/build-kernel-debian.yml +++ b/.github/workflows/build-kernel-debian.yml @@ -35,27 +35,27 @@ on: build: description: 'Name of this build, as listed in ci/build-matrix.yaml' type: string - default: 'qcom-next-trixie' + required: true flavour: description: 'Kernel flavour: the LOCALVERSION suffix and so the kernel release identity' type: string - default: 'qcom-next' + required: true suite: description: 'Target Debian suite (trixie, forky, sid)' type: string - default: 'trixie' + required: true kernel-branch: description: 'Source branch or pinned tag used by branch_tip or pinned_ref' type: string - default: 'qcom-next' + required: true git-clone: description: 'Kernel repository URL' type: string - default: 'https://github.com/qualcomm-linux/kernel' + required: true ref-strategy: description: 'Kernel ref strategy: latest_tag, branch_tip, or pinned_ref' type: string - default: 'latest_tag' + required: true tag-pattern: description: 'Tag glob used when ref-strategy=latest_tag' type: string @@ -63,11 +63,11 @@ on: srcpkg: description: 'Source package name' type: string - default: 'linux-qcom-next' + required: true binpkg: description: 'Binary metapackage name' type: string - default: 'linux-image-qcom-next' + required: true kernel-config: description: 'Extra config fragments applied on top of all of debian/config-available/' type: string @@ -79,7 +79,7 @@ on: debian-revision: description: 'Debian revision component of the package version' type: string - default: '0qli1' + required: true localversion: description: 'Override LOCALVERSION suffix (auto-derived from the resolved ref if empty)' type: string diff --git a/.github/workflows/build-kernel-ubuntu.yml b/.github/workflows/build-kernel-ubuntu.yml index c0c36423..c8dfe5e1 100644 --- a/.github/workflows/build-kernel-ubuntu.yml +++ b/.github/workflows/build-kernel-ubuntu.yml @@ -28,27 +28,27 @@ on: build: description: 'Name of this build, as listed in ci/build-matrix.yaml' type: string - default: 'qcom-next-resolute' + required: true flavour: description: 'Kernel flavour: the LOCALVERSION suffix and so the kernel release identity' type: string - default: 'qcom-next' + required: true suite: description: 'Target Ubuntu suite (noble, resolute)' type: string - default: 'resolute' + required: true kernel-branch: description: 'Source branch or pinned tag used by branch_tip or pinned_ref' type: string - default: 'qcom-next' + required: true git-clone: description: 'Kernel repository URL' type: string - default: 'https://github.com/qualcomm-linux/kernel' + required: true ref-strategy: description: 'Kernel ref strategy: latest_tag, branch_tip, or pinned_ref' type: string - default: 'latest_tag' + required: true tag-pattern: description: 'Tag glob used when ref-strategy=latest_tag' type: string @@ -56,11 +56,11 @@ on: srcpkg: description: 'Source package name' type: string - default: 'linux-qcom-next' + required: true binpkg: description: 'Binary metapackage name' type: string - default: 'linux-image-qcom-next' + required: true kernel-config: description: 'Extra config fragments applied on top of all of debian/config-available/' type: string @@ -72,7 +72,7 @@ on: debian-revision: description: 'Debian revision component of the package version' type: string - default: '0qli1' + required: true localversion: description: 'Override LOCALVERSION suffix (auto-derived from the resolved ref if empty)' type: string From f84dbd9f45025c672a792fa0d72b402008db667d Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:24 +0100 Subject: [PATCH 33/59] ci: build the source package in prepare prepare-kernel-source now runs build-source-package.sh after prepare-source.sh, in the same pkg-builder container, and uploads the resulting .changes set as source-package--. Nothing consumes it yet: the prepared-tree artifact is still uploaded and still what both build paths read, so this commit changes what a run produces and not what it builds. The next commits move each family onto it. The version the caller is told now comes from the fields file the script writes, read from the .dsc it built, rather than from a sed over the changelog on the runner host. The action also reports the orig tarball's name and SHA-256, which a caller can hold against what an archive already has for this upstream version. The container is told to trust the checkout, which the runner owns and root inside the container would otherwise refuse to read; it is this job's own clone. Signed-off-by: Christopher Obbard --- .../actions/prepare-kernel-source/action.yml | 95 +++++++++++++++---- 1 file changed, 74 insertions(+), 21 deletions(-) diff --git a/.github/actions/prepare-kernel-source/action.yml b/.github/actions/prepare-kernel-source/action.yml index 55adb59a..116c414d 100644 --- a/.github/actions/prepare-kernel-source/action.yml +++ b/.github/actions/prepare-kernel-source/action.yml @@ -1,7 +1,7 @@ name: Prepare kernel source description: > - Resolve the kernel ref, clone it, inject debian/ and generate the source - package, then upload the prepared tree as an artifact. + Resolve the kernel ref, clone it, inject debian/, build the Debian source + package, and upload it as an artifact alongside the prepared tree. # Every build takes this path, whichever family it belongs to, so it lives in a # composite action rather than a reusable workflow: the two family workflows @@ -21,13 +21,25 @@ description: > # come from the build matrix as the pull request wrote it. outputs: + srcpkg_name: + description: 'Name of the source package this build produces' + value: ${{ steps.version.outputs.srcpkg_name }} srcpkg_version: description: > - Version of the source package this build will produce, read from the - generated debian/changelog. A caller that promotes the result into an - archive needs it before the build starts, to see whether that version is - already published there. + Version of the source package this build produces, read from the .dsc + build-source-package.sh wrote. A caller that promotes the result into + an archive needs it before the build starts, to see whether that + version is already published there. value: ${{ steps.version.outputs.srcpkg_version }} + orig_file: + description: 'Filename of the .orig.tar.gz the source package references' + value: ${{ steps.version.outputs.orig_file }} + orig_sha256: + description: > + SHA-256 of that orig tarball. It is a function of the kernel commit, so + a caller can compare it with what an archive already holds for this + upstream version and fail on a difference. + value: ${{ steps.version.outputs.orig_sha256 }} inputs: build: @@ -352,28 +364,69 @@ runs: "ghcr.io/qualcomm-linux/pkg-builder:$DOCKER_DISTRO" \ ./prepare-source.sh "${ARGS[@]}" - - name: Read the generated source version - # prepare-source.sh has just written debian/changelog, so the version - # this build will produce is now decided and can be reported to the - # caller. Taken from the changelog rather than recomputed here: that file - # is what the source package is built from, so nothing can disagree with - # it. Parsed with sed rather than dpkg-parsechangelog because these steps - # run on the runner host, which is not required to have dpkg-dev; the - # first line of a changelog is "srcpkg (version) suite; urgency=...". - id: version + - name: Build source package + # The .orig.tar.gz, .dsc and .changes the binary build starts from, + # whichever family builds it. Built here, in the same container that + # ran prepare-source.sh, because the runner host has no dpkg-dev and + # because the orig's bytes depend on the gzip that wrote them: every + # build that must agree on a checksum writes its orig in this image. + # + # git is told to trust the checkout, which the runner owns and the + # container reads as root; the checkout is this job's own clone. The + # fields file lands beside the output directory, outside the artifact. shell: bash env: KERNEL_DIR: ${{ steps.clone.outputs.kernel_dir }} + DOCKER_DISTRO: ${{ steps.env.outputs.docker_distro }} + run: | + set -euo pipefail + docker run -i --rm \ + -v "$PWD:$PWD" --workdir="$PWD" \ + -e GIT_CONFIG_COUNT=1 \ + -e GIT_CONFIG_KEY_0=safe.directory -e GIT_CONFIG_VALUE_0='*' \ + "ghcr.io/qualcomm-linux/pkg-builder:$DOCKER_DISTRO" \ + ./build-source-package.sh \ + --source-dir "$KERNEL_DIR" \ + --output-dir "$PWD/source-package" \ + --write-fields "$PWD/source-package.fields" + + - name: Read the source package fields + # build-source-package.sh has decided the version and written it into + # the .dsc, so what it reports is what will be built and nothing can + # disagree with it. Read from its fields file rather than recomputed: + # these steps run on the runner host, which has no dpkg-dev. The values + # are the script's own, filtered to the characters Debian allows in a + # name and version, so none can carry a newline into GITHUB_OUTPUT. + id: version + shell: bash run: | set -euo pipefail - SRCPKG_VERSION=$(sed -n '1s/^[^ ]* (\([^)]*\)).*/\1/p' \ - "$KERNEL_DIR/debian/changelog") - [[ -n "$SRCPKG_VERSION" ]] || { - echo "::error::Could not read a version from the generated debian/changelog." + while IFS='=' read -r key value; do + case "$key" in + SRCPKG_NAME) echo "srcpkg_name=$value" ;; + SRCPKG_VERSION) echo "srcpkg_version=$value" ;; + ORIG) echo "orig_file=$(basename "$value")" ;; + ORIG_SHA256) echo "orig_sha256=$value" ;; + UPSTREAM_VERSION|CHANGES|DSC) ;; + *) echo "::error::unexpected field '$key' from build-source-package.sh"; exit 1 ;; + esac + done < source-package.fields >> "$GITHUB_OUTPUT" + grep -q '^SRCPKG_VERSION=.' source-package.fields || { + echo "::error::build-source-package.sh reported no version." exit 1 } - echo "Source version: $SRCPKG_VERSION" - echo "srcpkg_version=$SRCPKG_VERSION" >> "$GITHUB_OUTPUT" + echo "Source package:" + cat source-package.fields + + - name: Upload source package artifact + # The whole .changes set: .changes, .dsc, .debian.tar.xz and the orig. + # Plain files with no execute bits to lose, so no tarball around them. + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: source-package-${{ inputs.flavour }}-${{ inputs.suite }} + path: source-package/ + if-no-files-found: error + retention-days: 7 - name: Pack prepared source tree shell: bash From a2b82f478b37458c8ebde328bea0ea22d9cef8e8 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:24 +0100 Subject: [PATCH 34/59] ci: hand Debusine the source package prepare built The build job downloaded the prepared tree, unpacked it, and had debusine-action's generate-source-package assemble an orig tarball from it with tar czf -- a different tarball on every run. It now downloads the .changes set the prepare job built, reads the version from the .dsc, and gives lib/build that .dsc: the source package Debusine builds is the one prepare-kernel-source wrote and reported, with the reproducible orig. DEBUSINE_ASSEMBLE_ORIG in debusine-action stays for its other callers; nothing here uses it any more. Signed-off-by: Christopher Obbard --- .github/actions/debusine-build/action.yml | 46 +++++++++++++---------- .github/workflows/build-kernel-debian.yml | 4 +- 2 files changed, 28 insertions(+), 22 deletions(-) diff --git a/.github/actions/debusine-build/action.yml b/.github/actions/debusine-build/action.yml index 86a7404e..7e888b67 100644 --- a/.github/actions/debusine-build/action.yml +++ b/.github/actions/debusine-build/action.yml @@ -1,7 +1,7 @@ name: Build in Debusine description: > - Generate the Debian source package from the prepared tree and submit it to - Debusine, returning the CI workspace it was built in. + Submit the source package the prepare job built to Debusine, returning the + CI workspace it was built in. # Called by build-kernel-debian.yml, which decides what becomes of the result: # every build is downloaded and uploaded to S3, and one whose caller named a @@ -56,8 +56,8 @@ outputs: description: 'Browser URL of that workspace' value: ${{ steps.build-debusine.outputs.workspace_url }} srcpkg_version: - description: 'Version of the generated source package' - value: ${{ steps.generate-srcpkg.outputs.srcpkg_version }} + description: 'Version of the submitted source package' + value: ${{ steps.srcpkg.outputs.srcpkg_version }} runs: using: composite @@ -73,27 +73,33 @@ runs: sparse-checkout: | lib - - name: Download prepared source tree + # Into the workspace root: lib/build takes the .dsc from its working + # directory, and reaches its own helpers by a path relative to it, so the + # two have to be the same directory. + - name: Download source package uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 with: - name: kernel-srcpkg-${{ inputs.flavour }}-${{ inputs.suite }} - path: /tmp/srcpkg-artifact + name: source-package-${{ inputs.flavour }}-${{ inputs.suite }} - - name: Extract prepared source tree + - name: Read the source package version + # From the .dsc as downloaded, which is what Debusine will be given; + # the prepare job reported the same value from the same file. + id: srcpkg shell: bash run: | - mkdir srcpkg - tar xzf /tmp/srcpkg-artifact/*.tar.gz \ - -C srcpkg --strip-components=1 - - - name: Generate source package - id: generate-srcpkg - shell: bash - env: - SUITE: ${{ inputs.suite }} - DEBUSINE_ASSEMBLE_ORIG: "true" - run: | - SUITE="$SUITE" debusine-action/lib/generate-source-package + set -euo pipefail + DSC=$(ls -- *.dsc) + [[ $(wc -l <<<"$DSC") -eq 1 ]] || { + echo "::error::Expected exactly one .dsc in the source package artifact, found: $DSC" + exit 1 + } + SRCPKG_VERSION=$(sed -n 's/^Version: //p' "$DSC") + [[ "$SRCPKG_VERSION" =~ ^[A-Za-z0-9.+~:-]+$ ]] || { + echo "::error::Could not read a version from $DSC (got '$SRCPKG_VERSION')" + exit 1 + } + echo "Submitting $DSC ($SRCPKG_VERSION)" + echo "srcpkg_version=$SRCPKG_VERSION" >> "$GITHUB_OUTPUT" - name: Build in Debusine id: build-debusine diff --git a/.github/workflows/build-kernel-debian.yml b/.github/workflows/build-kernel-debian.yml index bcc0794c..ded15d08 100644 --- a/.github/workflows/build-kernel-debian.yml +++ b/.github/workflows/build-kernel-debian.yml @@ -22,7 +22,7 @@ name: build-kernel-debian # it releases rather than promoting a build some earlier run made. # # Jobs: -# prepare (self-hosted): clone, patch, inject debian/, generate the source, +# prepare (self-hosted): clone, inject debian/, build the source package, # and decide whether the version it produces is already published. # build (debusine-pkg-builder container): submit to Debusine. # publish (self-hosted): download the .deb files and upload them to S3. @@ -284,7 +284,7 @@ jobs: srcpkg_version: ${{ steps.debusine.outputs.srcpkg_version }} steps: # Only for the debusine-build action itself: the package being built - # comes from the prepared-source artifact, not from this checkout. + # comes from the source-package artifact, not from this checkout. - name: Checkout pkg-linux-qcom uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: From eb7c95f9fe1124d37d2383eed6ce8a385d603f76 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:24 +0100 Subject: [PATCH 35/59] ci: build the Ubuntu package from the source package The build job downloaded the prepared tree and ran build-kernel.sh --local-source --skip-prepare, which had docker_deb_build.py make an orig tarball of its own with tar czf before handing sbuild a .dsc. It now downloads the .changes set prepare built and passes its .dsc to build-kernel.sh --dsc, which runs sbuild on it in the suite's pkg-builder image. The Ubuntu family therefore builds the same kind of thing Debusine builds for the Debian family: the source package the run reported, with the reproducible orig, and nothing is prepared twice. Signed-off-by: Christopher Obbard --- .github/workflows/build-kernel-ubuntu.yml | 38 ++++++++++------------- 1 file changed, 16 insertions(+), 22 deletions(-) diff --git a/.github/workflows/build-kernel-ubuntu.yml b/.github/workflows/build-kernel-ubuntu.yml index c8dfe5e1..b6f7bce6 100644 --- a/.github/workflows/build-kernel-ubuntu.yml +++ b/.github/workflows/build-kernel-ubuntu.yml @@ -19,7 +19,7 @@ name: build-kernel-ubuntu # in ci/build-matrix.yaml, rather than in a dispatch form that drifts from it. # # Jobs: -# prepare (self-hosted): clone, patch, inject debian/, generate the source. +# prepare (self-hosted): clone, inject debian/, build the source package. # build (self-hosted): build the .deb files and upload them to S3. on: @@ -122,7 +122,7 @@ jobs: kver-extra: ${{ inputs.kver-extra }} # --------------------------------------------------------------------------- - # build: build the prepared source in a suite-matched container, then publish. + # build: build the source package in a suite-matched container, then publish. # # Named for where the build happens, as the Debian workflows' Build package # (Debusine) is: this one runs in a suite-matched pkg-builder container on @@ -178,34 +178,28 @@ jobs: esac docker image ls - - name: Download prepared source tree + - name: Download source package uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 with: - name: kernel-srcpkg-${{ inputs.flavour }}-${{ inputs.suite }} - path: /tmp/srcpkg-artifact - - - name: Extract prepared source tree - run: | - # tar.gz preserves the execute bits that upload-artifact's zip would - # strip (see README); --strip-components=1 drops the archive's top dir. - mkdir -p kernel-source - tar xzf /tmp/srcpkg-artifact/*.tar.gz \ - -C kernel-source --strip-components=1 - echo "KERNEL_DIR=$GITHUB_WORKSPACE/kernel-source" >> "$GITHUB_ENV" + name: source-package-${{ inputs.flavour }}-${{ inputs.suite }} + path: source-package - name: Build kernel package run: | - # Kernel source was cloned, patched, and prepared by the prepare job. - # Pass --local-source and --skip-prepare so build-kernel.sh goes - # directly to the build step. debian/control, debian/changelog, and - # all config fragments are already baked into the extracted source - # tree; no re-derivation of LOCALVERSION, kver-extra, or debug config - # is needed here. + # The prepare job built the source package; --dsc hands it to + # sbuild in the suite's pkg-builder image, so what is built here + # is the .dsc the run reported, exactly as Debusine builds it for + # the Debian family. Nothing is cloned or prepared again. + set -euo pipefail + DSC=$(ls source-package/*.dsc) + [[ $(wc -l <<<"$DSC") -eq 1 ]] || { + echo "::error::Expected exactly one .dsc in the source package artifact, found: $DSC" + exit 1 + } ./build-kernel.sh \ --build-mode docker \ --distro "$DISTRO" \ - --local-source "$KERNEL_DIR" \ - --skip-prepare \ + --dsc "$DSC" \ --docker-build "$GITHUB_WORKSPACE/docker-pkg-build/docker_deb_build.py" - name: Upload .deb packages to S3 From bf49e2625e01e15704df41ba68bdffb66e77981d Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:24 +0100 Subject: [PATCH 36/59] ci: stop uploading the prepared tree Both build paths now start from the source-package artifact, so the tarball of the prepared tree has no reader. It was the larger of the two uploads, and the reason the README explained why an artifact had to be a tar.gz: the tree carried execute bits that upload-artifact's zip would strip. A .changes set is plain files, so that note goes with it, and the pipeline diagrams now show the source package as the handoff. Signed-off-by: Christopher Obbard --- .../actions/prepare-kernel-source/action.yml | 19 +--------- README.md | 35 ++++++++++--------- 2 files changed, 20 insertions(+), 34 deletions(-) diff --git a/.github/actions/prepare-kernel-source/action.yml b/.github/actions/prepare-kernel-source/action.yml index 116c414d..fe6dd19d 100644 --- a/.github/actions/prepare-kernel-source/action.yml +++ b/.github/actions/prepare-kernel-source/action.yml @@ -1,7 +1,7 @@ name: Prepare kernel source description: > Resolve the kernel ref, clone it, inject debian/, build the Debian source - package, and upload it as an artifact alongside the prepared tree. + package, and upload it as an artifact for the build job to consume. # Every build takes this path, whichever family it belongs to, so it lives in a # composite action rather than a reusable workflow: the two family workflows @@ -428,20 +428,3 @@ runs: if-no-files-found: error retention-days: 7 - - name: Pack prepared source tree - shell: bash - env: - KERNEL_DIR: ${{ steps.clone.outputs.kernel_dir }} - run: | - set -euo pipefail - tar czf /tmp/kernel-srcpkg.tar.gz \ - -C "$(dirname "$KERNEL_DIR")" \ - "$(basename "$KERNEL_DIR")" - - - name: Upload prepared source tree artifact - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 - with: - name: kernel-srcpkg-${{ inputs.flavour }}-${{ inputs.suite }} - path: /tmp/kernel-srcpkg.tar.gz - if-no-files-found: error - retention-days: 7 diff --git a/README.md b/README.md index ab2dcf8c..5d1986b3 100644 --- a/README.md +++ b/README.md @@ -299,7 +299,7 @@ The build workflows pass their entry's own `debian_revision` through, and the `prepare-kernel-source` action requires it: there is no caller without an entry in hand, so there is no fallback to look one up. -Each entry has a distinct prepared-source artifact, Debusine child workspace, +Each entry has a distinct source-package artifact, Debusine child workspace, and S3 path keyed by `flavour + suite`. This prevents two flavours that both build, for example, `trixie` from consuming or publishing each other's inputs or outputs. They key on `flavour`, not on the build's `name`, so renaming a @@ -417,24 +417,29 @@ flowchart LR K["Matrix-selected kernel repository\nlatest tag, branch tip, or pinned ref"] --> PS M["pkg-linux-qcom\ndebian/ and ci/ from this commit"] --> PS - PS["prepare-source.sh\n\nInject debian/\nApply all config-available fragments plus any extras\nGenerate control, changelog, localversion, pkgversion"] --> TAR - TAR["tar czf kernel-srcpkg-variant-suite.tar.gz\nPreserves execute permissions"] --> ART - ART["GitHub Actions artifact\nOne prepared source tree per variant + suite"] + PS["prepare-source.sh\n\nInject debian/\nApply all config-available fragments plus any extras\nGenerate control, changelog, localversion, pkgversion"] --> BSP + BSP["build-source-package.sh\n\ngit archive the commit → .orig.tar.gz\ndpkg-source -b → .dsc, .debian.tar.xz\ndpkg-genchanges -S → .changes"] --> ART + ART["GitHub Actions artifact\nsource-package-variant-suite\nOne .changes set per variant + suite"] ``` -> **Why `tar.gz`?** `actions/upload-artifact` uses zip internally, which strips -> Unix execute bits. Kernel build scripts require those permissions. The tar -> archive preserves them between the prepare and build jobs. +Both steps run in the pkg-builder container on the self-hosted runner. The +orig tarball is a function of the kernel commit: `git archive` gives every +entry the commit's timestamp and root ownership, and `gzip -n` writes no +timestamp, so two runs on one commit write one tarball and two suites that +differ only in Debian revision share it. `build-source-package.sh` checks +that the commit it archives is the one the version names (`~g`) and +that the tree matches that commit outside `debian/`. See +[debian/README.md](debian/README.md#building-a-source-package) for the +local equivalent. ### Debian build path ```mermaid flowchart LR - ART["kernel-srcpkg-variant-suite\nartifact"] --> GSP + ART["source-package-variant-suite\nartifact"] --> SUBMIT subgraph source[GitHub build job: debusine-pkg-builder container] - GSP["generate-source-package\nDEBUSINE_ASSEMBLE_ORIG=true\n\nCreate .orig.tar.gz\nRun dpkg-buildpackage -S\nProduce .dsc"] --> SUBMIT - SUBMIT["lib/build\nCreate CI child workspace\nSubmit source package to Debusine"] + SUBMIT["lib/build\nCreate CI child workspace\nSubmit the .dsc to Debusine"] end SUBMIT --> DEB["Debusine\nBuild binary packages"] @@ -470,19 +475,17 @@ flowchart LR ```mermaid flowchart LR - ART["kernel-srcpkg-variant-suite\nartifact"] --> EXT + ART["source-package-variant-suite\nartifact"] --> BK subgraph build[Ubuntu build job] - EXT["Extract prepared source tree\n--strip-components=1"] --> BK - BK["build-kernel.sh\n--skip-prepare\n--local-source\n--build-mode docker\ndpkg-buildpackage -b"] --> S3 + BK["build-kernel.sh --dsc\n--build-mode docker\nsbuild in the suite's pkg-builder image"] --> S3 end S3["S3\npackage artifacts"] ``` -`--skip-prepare` is safe because `prepare-source.sh` has already generated the -packaging metadata and applied the config fragments before the artifact is -created. +Nothing is cloned or prepared here: the `.dsc` the prepare job reported is +what sbuild builds, as it is what Debusine builds on the Debian path. ## Packages From d0914ea13fdb4cdc7d720aa35da878f65e0678a7 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:24 +0100 Subject: [PATCH 37/59] ci: fail when the archive holds a different orig for this version The orig tarball is now a function of the kernel commit, so the archive can hold a different one for the same upstream version only if something about how it was written differs -- and that is what this check is for. The already-published step, which already reads the target workspace's source index, now also finds every stanza naming this run's orig and compares its SHA-256 with the one prepare reported. A mismatch fails the run with both checksums, before anything is built, rather than surfacing as a file conflict at promotion or, worse, not at all. The case that matters is the packaging rebuild: 0qli1 to 0qli2 of one snapshot, where the archive requires the two revisions to share one file. The check reads this suite's index only, since apt indexes are per suite; across suites the reproducibility of the tarball itself is the guarantee. Signed-off-by: Christopher Obbard --- .github/workflows/build-kernel-debian.yml | 30 +++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build-kernel-debian.yml b/.github/workflows/build-kernel-debian.yml index ded15d08..782b2ccb 100644 --- a/.github/workflows/build-kernel-debian.yml +++ b/.github/workflows/build-kernel-debian.yml @@ -224,6 +224,8 @@ jobs: SUITE: ${{ inputs.suite }} SRCPKG: ${{ inputs.srcpkg }} SRCPKG_VERSION: ${{ steps.prepare.outputs.srcpkg_version }} + ORIG_FILE: ${{ steps.prepare.outputs.orig_file }} + ORIG_SHA256: ${{ steps.prepare.outputs.orig_sha256 }} run: | # No -x: DEBUSINE_TOKEN is in env, keep xtrace off. set -euo pipefail @@ -247,8 +249,32 @@ jobs: exit 0 fi - if chdist apt-cache target-workspace showsrc "$SRCPKG" 2>/dev/null \ - | grep -qxF "Version: $SRCPKG_VERSION"; then + SOURCES=$(chdist apt-cache target-workspace showsrc "$SRCPKG" 2>/dev/null || true) + + # The orig this run wrote is a function of the kernel commit, and + # the archive may hold it already: under this version on a rerun, + # or under another Debian revision of the same upstream version -- + # a packaging rebuild, which is exactly the case where the archive + # requires the two revisions to share one file. Every stanza that + # names this orig must therefore carry this checksum. One that does + # not is a build that would be rejected at promotion for a file + # conflict, or worse, accepted somewhere that does not check; so it + # fails here, with both checksums, before anything is built. + # + # This suite only: apt indexes are per suite, so an orig published + # under another suite of the workspace is not seen here. The + # reproducibility of the tarball is what covers that case. + ARCHIVE_SHA256=$(awk -v orig="$ORIG_FILE" \ + '$1 ~ /^[0-9a-f]{64}$/ && $3 == orig { print $1 }' <<<"$SOURCES" | sort -u) + if [[ -n "$ARCHIVE_SHA256" && "$ARCHIVE_SHA256" != "$ORIG_SHA256" ]]; then + echo "::error::$DEBUSINE_WORKSPACE $SUITE already holds $ORIG_FILE with sha256 $ARCHIVE_SHA256, but this run built it as $ORIG_SHA256." + echo "::error::An orig tarball is a function of the commit it names; find what differs between the two builds before publishing either." + exit 1 + fi + [[ -z "$ARCHIVE_SHA256" ]] \ + || echo "$ORIG_FILE in $DEBUSINE_WORKSPACE $SUITE matches this build ($ORIG_SHA256)." + + if grep -qxF "Version: $SRCPKG_VERSION" <<<"$SOURCES"; then echo "$SRCPKG $SRCPKG_VERSION is already in $DEBUSINE_WORKSPACE." echo "Nothing new to build; the rest of this run is skipped." \ >> "$GITHUB_STEP_SUMMARY" From 202d9e9b34dc48d46edb1e3cb18a7f7cf8dad846 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:24 +0100 Subject: [PATCH 38/59] ci: build the Ubuntu leg in one job The prepare job and the build job ran on the same runner label, so the only thing the artifact between them did was cross a job boundary that had no other reason to exist: uploaded by one ephemeral runner to be downloaded by the next. One job now clones, prepares, builds the source package and builds the binaries from it in place. The action still uploads the .changes set, as the record of what was built, but nothing downloads it. The build job's own image-fetch step goes with it. It repeated the action's pull-or-build decision for one suite; the action makes that decision for every suite before prepare-source.sh runs, and leaves the docker-pkg-build checkout where build-kernel.sh looks for it when it had to build the image. Signed-off-by: Christopher Obbard --- .github/workflows/build-kernel-ubuntu.yml | 107 +++++++--------------- README.md | 8 +- 2 files changed, 35 insertions(+), 80 deletions(-) diff --git a/.github/workflows/build-kernel-ubuntu.yml b/.github/workflows/build-kernel-ubuntu.yml index b6f7bce6..553297d1 100644 --- a/.github/workflows/build-kernel-ubuntu.yml +++ b/.github/workflows/build-kernel-ubuntu.yml @@ -18,9 +18,8 @@ name: build-kernel-ubuntu # per Ubuntu entry it selected. That also keeps every value describing a build # in ci/build-matrix.yaml, rather than in a dispatch form that drifts from it. # -# Jobs: -# prepare (self-hosted): clone, inject debian/, build the source package. -# build (self-hosted): build the .deb files and upload them to S3. +# One job, on the self-hosted runner: clone, inject debian/, build the source +# package, build the .deb files from it, and upload them to S3. on: workflow_call: @@ -88,20 +87,38 @@ permissions: jobs: # --------------------------------------------------------------------------- - # prepare: clone kernel source, inject packaging, run prepare-source.sh. + # build: clone, prepare, build the source package, build binaries from it, + # and publish them. + # + # One job rather than a prepare job and a build job, because every step + # runs on the same runner label: the source package the prepare action + # writes is consumed here from the workspace it was written into, with no + # artifact uploaded only to be downloaded again by the next job. The action + # still uploads it, as the record of what was built. + # + # Named for where the build happens, as the Debian workflow's Build package + # (Debusine) is: this one runs in a suite-matched pkg-builder container on + # the runner rather than being submitted to a build service. # --------------------------------------------------------------------------- - prepare: - name: Generate source package + build: + name: Build package (Docker) runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] permissions: contents: read # actions/checkout packages: read # docker pull ghcr.io/qualcomm-linux/pkg-builder + env: + DISTRO: ${{ inputs.suite }} + defaults: + run: + shell: bash steps: - name: Checkout pkg-linux-qcom uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false + # Also pulls, or builds from a docker-pkg-build checkout, the suite's + # pkg-builder image -- the image the binary build below runs sbuild in. - name: Prepare kernel source uses: ./.github/actions/prepare-kernel-source with: @@ -121,86 +138,24 @@ jobs: localversion: ${{ inputs.localversion }} kver-extra: ${{ inputs.kver-extra }} - # --------------------------------------------------------------------------- - # build: build the source package in a suite-matched container, then publish. - # - # Named for where the build happens, as the Debian workflows' Build package - # (Debusine) is: this one runs in a suite-matched pkg-builder container on - # the runner rather than being submitted to a build service. - # --------------------------------------------------------------------------- - build: - name: Build package (Docker) - needs: prepare - runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] - permissions: - contents: read # actions/checkout - packages: read # docker pull ghcr.io/qualcomm-linux/pkg-builder - env: - DISTRO: ${{ inputs.suite }} - defaults: - run: - shell: bash - steps: - - name: Checkout pkg-linux-qcom - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - persist-credentials: false - - # Pinned to @main while native kernel-build support is still landing - # upstream in qualcomm-linux/docker-pkg-build. - - name: Checkout docker-pkg-build - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - persist-credentials: false - repository: qualcomm-linux/docker-pkg-build - ref: main - path: docker-pkg-build - - - name: Fetch builder image - env: - REGISTRY_TOKEN: ${{ secrets.GITHUB_TOKEN }} - run: | - # No -x: REGISTRY_TOKEN is in env. - set -euo pipefail - # Same split as prepare-kernel-source: pull what docker-pkg-build - # publishes, build what it does not. Nothing is forced here, because - # docker_deb_build.py builds the image itself when the build below - # finds it missing -- so an unpublished suite needs no step of its - # own, only the absence of a pull. - case "$DISTRO" in - resolute) - echo "$REGISTRY_TOKEN" | docker login ghcr.io -u "$GITHUB_ACTOR" --password-stdin - docker pull "ghcr.io/qualcomm-linux/pkg-builder:$DISTRO" - ;; - *) - echo "No published pkg-builder image for $DISTRO; it will be built on demand." - ;; - esac - docker image ls - - - name: Download source package - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 - with: - name: source-package-${{ inputs.flavour }}-${{ inputs.suite }} - path: source-package - - name: Build kernel package run: | - # The prepare job built the source package; --dsc hands it to - # sbuild in the suite's pkg-builder image, so what is built here - # is the .dsc the run reported, exactly as Debusine builds it for - # the Debian family. Nothing is cloned or prepared again. + # The source package sits where the action wrote it. --dsc hands + # it to sbuild in the suite's pkg-builder image, so what is built + # here is the .dsc the run reported, exactly as Debusine builds it + # for the Debian family. Nothing is cloned or prepared again, and + # build-kernel.sh finds the docker-pkg-build checkout the action + # leaves behind when it had to build the image itself. set -euo pipefail DSC=$(ls source-package/*.dsc) [[ $(wc -l <<<"$DSC") -eq 1 ]] || { - echo "::error::Expected exactly one .dsc in the source package artifact, found: $DSC" + echo "::error::Expected exactly one .dsc in source-package/, found: $DSC" exit 1 } ./build-kernel.sh \ --build-mode docker \ --distro "$DISTRO" \ - --dsc "$DSC" \ - --docker-build "$GITHUB_WORKSPACE/docker-pkg-build/docker_deb_build.py" + --dsc "$DSC" - name: Upload .deb packages to S3 # build-kernel.sh outputs to kernel-build// by default. diff --git a/README.md b/README.md index 5d1986b3..989f2220 100644 --- a/README.md +++ b/README.md @@ -377,9 +377,9 @@ flowchart TD end subgraph build[One build workflow per leg] - C2["prepare\nprepare-kernel-source action\nClone ref, run prepare-source.sh\nSkip the run if the version is published"] + C2["prepare\nprepare-kernel-source action\nClone ref, build the source package\nSkip the run if the version is published"] C3["build\ndebusine-build action"] - C4["build\nbuild-kernel.sh in Docker"] + C4["build\nprepare-kernel-source action, then\nbuild-kernel.sh --dsc in Docker"] C5["publish\nDownload .deb files, upload to S3"] C6["promote\nlib/release into the caller's workspace"] end @@ -396,8 +396,8 @@ flowchart TD A5 --> B1 B1 --> B2 & B5 B2 --> C2 - B5 --> C2 - C2 --> C3 & C4 + B5 --> C4 + C2 --> C3 C3 --> C5 & C6 C4 --> D1 C5 --> D1 From b472673b1d889f564c1b22c79b6b0eaf076de751 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:24 +0100 Subject: [PATCH 39/59] ci: publish the source package to S3 beside the binaries The .deb files reach S3 under one run's directory; the source package they were built from now sits in source/ beneath it, on both paths. The .changes records every checksum, the .orig.tar.gz is the exact source, and the .dsc is enough to rebuild the packages anywhere. The Actions artifact holding the same files expires after a week; this is the copy that stays with the binaries. The Debian publish job downloads the artifact for it, since the source package was built on another runner; the Ubuntu job uploads it from the workspace it was built in. Signed-off-by: Christopher Obbard --- .github/workflows/build-kernel-debian.yml | 17 +++++++++++++++++ .github/workflows/build-kernel-ubuntu.yml | 10 ++++++++++ README.md | 4 ++++ 3 files changed, 31 insertions(+) diff --git a/.github/workflows/build-kernel-debian.yml b/.github/workflows/build-kernel-debian.yml index 782b2ccb..cafecfac 100644 --- a/.github/workflows/build-kernel-debian.yml +++ b/.github/workflows/build-kernel-debian.yml @@ -396,6 +396,23 @@ jobs: path: deb-artifacts destination: ${{ github.repository_owner }}/pkg/debusine/${{ github.event.repository.name }}/${{ inputs.flavour }}/${{ inputs.suite }}/${{ github.run_id }}-${{ github.run_attempt }}/ + # The source package the binaries were built from, kept beside them: + # the .changes records every checksum, and the .dsc is enough to + # rebuild the packages anywhere. The Actions artifact of the same + # files expires; this does not. + - name: Download source package + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: source-package-${{ inputs.flavour }}-${{ inputs.suite }} + path: source-package + + - name: Upload source package to S3 + uses: qualcomm-linux/upload-private-artifact-action@4940e9327cd7386acdb908b61c714c079b7d754b # aws-v4 + with: + s3_bucket: ${{ vars.ARTIFACT_S3_BUCKET }} + path: source-package + destination: ${{ github.repository_owner }}/pkg/debusine/${{ github.event.repository.name }}/${{ inputs.flavour }}/${{ inputs.suite }}/${{ github.run_id }}-${{ github.run_attempt }}/source/ + # --------------------------------------------------------------------------- # promote: copy the built package from the ephemeral CI workspace into the # workspace the caller named, making it installable from that archive. diff --git a/.github/workflows/build-kernel-ubuntu.yml b/.github/workflows/build-kernel-ubuntu.yml index 553297d1..20b2acf9 100644 --- a/.github/workflows/build-kernel-ubuntu.yml +++ b/.github/workflows/build-kernel-ubuntu.yml @@ -167,3 +167,13 @@ jobs: s3_bucket: ${{ vars.ARTIFACT_S3_BUCKET }} path: kernel-build/${{ inputs.suite }} destination: ${{ github.repository_owner }}/pkg/temp/${{ github.event.repository.name }}/${{ inputs.flavour }}/${{ inputs.suite }}/${{ github.run_id }}-${{ github.run_attempt }}/ + + # The source package the binaries were built from, kept beside them, + # as the Debian workflow keeps its own. Still in the workspace where + # the action wrote it. + - name: Upload source package to S3 + uses: qualcomm-linux/upload-private-artifact-action@4940e9327cd7386acdb908b61c714c079b7d754b # aws-v4 + with: + s3_bucket: ${{ vars.ARTIFACT_S3_BUCKET }} + path: source-package + destination: ${{ github.repository_owner }}/pkg/temp/${{ github.event.repository.name }}/${{ inputs.flavour }}/${{ inputs.suite }}/${{ github.run_id }}-${{ github.run_attempt }}/source/ diff --git a/README.md b/README.md index 989f2220..f39aea73 100644 --- a/README.md +++ b/README.md @@ -313,6 +313,10 @@ S3 outputs use these layouts, where `` is /pkg/temp///// ``` +Each holds the `.deb` files, and a `source/` directory beneath it holds the +source package they were built from: the `.changes`, `.dsc`, `.debian.tar.xz` +and `.orig.tar.gz`. + The first layout is for Debian/Debusine builds; the second is for Ubuntu Docker builds. Consumers must select the intended flavour and suite. From da2c771874babe936ca54571cef4935c447905c9 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 02:29:24 +0100 Subject: [PATCH 40/59] ci: test for a published version before comparing the orig The orig comparison ran first, so a night on which the tracked tag had not moved failed instead of skipping: the version was already published, and the orig in the archive predates git archive, so the checksums differ. That is a run with nothing to do reported as a conflict. A run that publishes nothing uploads no files, so nothing it built can conflict with anything. The already-published test therefore comes first and exits, and the orig comparison guards only a version that is actually going to be promoted. That leaves it doing the job it was added for: a packaging rebuild publishing a second Debian revision of one upstream version, where the archive requires both revisions to name one file. Signed-off-by: Christopher Obbard --- .github/workflows/build-kernel-debian.yml | 42 +++++++++++++---------- 1 file changed, 23 insertions(+), 19 deletions(-) diff --git a/.github/workflows/build-kernel-debian.yml b/.github/workflows/build-kernel-debian.yml index cafecfac..715d154e 100644 --- a/.github/workflows/build-kernel-debian.yml +++ b/.github/workflows/build-kernel-debian.yml @@ -251,15 +251,29 @@ jobs: SOURCES=$(chdist apt-cache target-workspace showsrc "$SRCPKG" 2>/dev/null || true) - # The orig this run wrote is a function of the kernel commit, and - # the archive may hold it already: under this version on a rerun, - # or under another Debian revision of the same upstream version -- - # a packaging rebuild, which is exactly the case where the archive - # requires the two revisions to share one file. Every stanza that - # names this orig must therefore carry this checksum. One that does - # not is a build that would be rejected at promotion for a file - # conflict, or worse, accepted somewhere that does not check; so it - # fails here, with both checksums, before anything is built. + # Whether there is anything to do at all, before anything is + # inspected about what would be built: a run that publishes nothing + # uploads no files, so nothing it holds can conflict with the + # archive. + if grep -qxF "Version: $SRCPKG_VERSION" <<<"$SOURCES"; then + echo "$SRCPKG $SRCPKG_VERSION is already in $DEBUSINE_WORKSPACE." + echo "Nothing new to build; the rest of this run is skipped." \ + >> "$GITHUB_STEP_SUMMARY" + echo "already_published=true" >> "$GITHUB_OUTPUT" + exit 0 + fi + echo "$SRCPKG $SRCPKG_VERSION is not in $DEBUSINE_WORKSPACE yet." + echo "already_published=false" >> "$GITHUB_OUTPUT" + + # This version is going to be built and promoted, so the orig it + # carries has to agree with any the archive already holds under that + # name. It can hold one: a packaging rebuild publishes a second + # Debian revision of one upstream version, and the archive requires + # the two revisions to share the file. The orig is a function of the + # kernel commit, so a stanza naming it with another checksum means + # two builds of one commit disagreed. That is rejected at promotion + # as a file conflict, or worse accepted somewhere that does not + # check, so it fails here with both checksums instead. # # This suite only: apt indexes are per suite, so an orig published # under another suite of the workspace is not seen here. The @@ -274,16 +288,6 @@ jobs: [[ -z "$ARCHIVE_SHA256" ]] \ || echo "$ORIG_FILE in $DEBUSINE_WORKSPACE $SUITE matches this build ($ORIG_SHA256)." - if grep -qxF "Version: $SRCPKG_VERSION" <<<"$SOURCES"; then - echo "$SRCPKG $SRCPKG_VERSION is already in $DEBUSINE_WORKSPACE." - echo "Nothing new to build; the rest of this run is skipped." \ - >> "$GITHUB_STEP_SUMMARY" - echo "already_published=true" >> "$GITHUB_OUTPUT" - else - echo "$SRCPKG $SRCPKG_VERSION is not in $DEBUSINE_WORKSPACE yet." - echo "already_published=false" >> "$GITHUB_OUTPUT" - fi - # --------------------------------------------------------------------------- # build: generate the Debian source package and submit it to Debusine. # From ce5dbe24c8f8a7a14d686a08d7893bf6dd63ece8 Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Sat, 5 Sep 2026 03:07:45 +0100 Subject: [PATCH 41/59] ci: inline the debusine-build action into its one caller The action had a single caller and paid for being one. The vars and secrets contexts do not reach a composite action, so the Debusine host, scope, user and token were all declared as inputs whose only purpose was to carry values the calling workflow could already see. Its steps now sit in the build job and read vars and secrets directly, and six inputs and their plumbing go. Two more things fall out. The job checked out pkg-linux-qcom solely to reach the action, so that checkout goes too: the package comes from the artifact and the tooling from debusine-action, and nothing of this repository is needed in that job at all. And workspace_url was an action output that only ever fed a step summary, so it is now a local variable in the step that writes it. Signed-off-by: Christopher Obbard --- .github/actions/debusine-build/action.yml | 131 ---------------------- .github/workflows/build-kernel-debian.yml | 79 ++++++++++--- README.md | 14 +-- 3 files changed, 70 insertions(+), 154 deletions(-) delete mode 100644 .github/actions/debusine-build/action.yml diff --git a/.github/actions/debusine-build/action.yml b/.github/actions/debusine-build/action.yml deleted file mode 100644 index 7e888b67..00000000 --- a/.github/actions/debusine-build/action.yml +++ /dev/null @@ -1,131 +0,0 @@ -name: Build in Debusine -description: > - Submit the source package the prepare job built to Debusine, returning the - CI workspace it was built in. - -# Called by build-kernel-debian.yml, which decides what becomes of the result: -# every build is downloaded and uploaded to S3, and one whose caller named a -# target workspace is also promoted into it. Holding these steps in a composite -# action keeps the submission itself in one place, separate from the job that -# declares the runner, container and environment it happens in. -# -# The caller must have checked out pkg-linux-qcom into the workspace first, -# which is also where this action comes from. Runs inside the -# debusine-pkg-builder container. -# -# vars and secrets contexts do not reach a composite action, so the Debusine -# host, scope and credentials come in as inputs from the calling workflow. - -inputs: - suite: - description: 'Target Debian suite (trixie, forky, sid)' - required: true - flavour: - description: 'Kernel flavour, isolating artifacts and Debusine workspaces' - required: true - debusine-host: - description: 'Debusine instance hostname' - required: true - debusine-scope: - description: 'Debusine scope' - required: true - debusine-user: - description: 'Debusine account used to submit the build' - required: true - debusine-token: - description: 'Debusine token for the CI child workspace' - required: true - debusine-parent-workspace: - description: 'Parent Debusine workspace for CI child workspace creation' - default: 'qli-ci' - extra-build-dep-workspaces: - description: > - Workspaces the build resolves its build-dependencies from, space - separated. Set by the calling workflow, which knows whether this build is - one a release may promote. - default: 'qli' - debusine-action-ref: - description: 'Ref of qualcomm-linux/debusine-action to use' - default: 'main' - -outputs: - workspace: - description: 'Debusine CI workspace the package was built in' - value: ${{ steps.build-debusine.outputs.workspace }} - workspace_url: - description: 'Browser URL of that workspace' - value: ${{ steps.build-debusine.outputs.workspace_url }} - srcpkg_version: - description: 'Version of the submitted source package' - value: ${{ steps.srcpkg.outputs.srcpkg_version }} - -runs: - using: composite - steps: - - name: Checkout debusine-action helpers - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - persist-credentials: false - repository: qualcomm-linux/debusine-action - ref: ${{ inputs.debusine-action-ref }} - path: debusine-action - fetch-depth: 1 - sparse-checkout: | - lib - - # Into the workspace root: lib/build takes the .dsc from its working - # directory, and reaches its own helpers by a path relative to it, so the - # two have to be the same directory. - - name: Download source package - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 - with: - name: source-package-${{ inputs.flavour }}-${{ inputs.suite }} - - - name: Read the source package version - # From the .dsc as downloaded, which is what Debusine will be given; - # the prepare job reported the same value from the same file. - id: srcpkg - shell: bash - run: | - set -euo pipefail - DSC=$(ls -- *.dsc) - [[ $(wc -l <<<"$DSC") -eq 1 ]] || { - echo "::error::Expected exactly one .dsc in the source package artifact, found: $DSC" - exit 1 - } - SRCPKG_VERSION=$(sed -n 's/^Version: //p' "$DSC") - [[ "$SRCPKG_VERSION" =~ ^[A-Za-z0-9.+~:-]+$ ]] || { - echo "::error::Could not read a version from $DSC (got '$SRCPKG_VERSION')" - exit 1 - } - echo "Submitting $DSC ($SRCPKG_VERSION)" - echo "srcpkg_version=$SRCPKG_VERSION" >> "$GITHUB_OUTPUT" - - - name: Build in Debusine - id: build-debusine - shell: bash - env: - GITHUB_REPOSITORY_ID: ${{ github.repository_id }} - GITHUB_RUN_ID: ${{ github.run_id }} - GITHUB_RUN_ATTEMPT: ${{ github.run_attempt }} - JOB_INDEX: ${{ inputs.flavour }}-${{ inputs.suite }} - DEBUSINE_HOST: ${{ inputs.debusine-host }} - DEBUSINE_SCOPE: ${{ inputs.debusine-scope }} - DEBUSINE_USER: ${{ inputs.debusine-user }} - DEBUSINE_TOKEN: ${{ inputs.debusine-token }} - DEBUSINE_PARENT_WORKSPACE: ${{ inputs.debusine-parent-workspace }} - SUITE: ${{ inputs.suite }} - EXTRA_BUILD_DEP_WORKSPACES: ${{ inputs.extra-build-dep-workspaces }} - run: | - # No -x here: DEBUSINE_TOKEN is in env, keep xtrace off. - set -euo pipefail - debusine-action/lib/build - workspace=$(sed -n 's/^workspace=//p' "$GITHUB_OUTPUT") - echo "workspace_url=https://${DEBUSINE_HOST}/${DEBUSINE_SCOPE}/${workspace}/" >> "$GITHUB_OUTPUT" - - - name: Note Debusine workspace URL - shell: bash - env: - WORKSPACE_URL: ${{ steps.build-debusine.outputs.workspace_url }} - run: | - echo "Debusine Workspace URL: $WORKSPACE_URL" >> "$GITHUB_STEP_SUMMARY" diff --git a/.github/workflows/build-kernel-debian.yml b/.github/workflows/build-kernel-debian.yml index 715d154e..4123c042 100644 --- a/.github/workflows/build-kernel-debian.yml +++ b/.github/workflows/build-kernel-debian.yml @@ -289,7 +289,11 @@ jobs: || echo "$ORIG_FILE in $DEBUSINE_WORKSPACE $SUITE matches this build ($ORIG_SHA256)." # --------------------------------------------------------------------------- - # build: generate the Debian source package and submit it to Debusine. + # build: submit the source package prepare built to Debusine, and wait. + # + # Nothing of this repository is needed here: the package comes from the + # source-package artifact and the tooling from debusine-action, so this job + # checks out neither pkg-linux-qcom nor anything else of ours. # # Skipped, with everything after it, when prepare found this version already # published in the target workspace. @@ -311,27 +315,72 @@ jobs: password: ${{ secrets.GITHUB_TOKEN }} outputs: workspace: ${{ steps.debusine.outputs.workspace }} - srcpkg_version: ${{ steps.debusine.outputs.srcpkg_version }} + srcpkg_version: ${{ steps.srcpkg.outputs.srcpkg_version }} + defaults: + run: + shell: bash steps: - # Only for the debusine-build action itself: the package being built - # comes from the source-package artifact, not from this checkout. - - name: Checkout pkg-linux-qcom + - name: Checkout debusine-action helpers uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false + repository: qualcomm-linux/debusine-action + ref: main + path: debusine-action + fetch-depth: 1 + sparse-checkout: | + lib + + # Into the workspace root: lib/build takes the .dsc from its working + # directory, and reaches its own helpers by a path relative to it, so the + # two have to be the same directory. + - name: Download source package + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: source-package-${{ inputs.flavour }}-${{ inputs.suite }} + + - name: Read the source package version + # From the .dsc as downloaded, which is what Debusine will be given; + # the prepare job reported the same value from the same file. + id: srcpkg + run: | + set -euo pipefail + DSC=$(ls -- *.dsc) + [[ $(wc -l <<<"$DSC") -eq 1 ]] || { + echo "::error::Expected exactly one .dsc in the source package artifact, found: $DSC" + exit 1 + } + SRCPKG_VERSION=$(sed -n 's/^Version: //p' "$DSC") + [[ "$SRCPKG_VERSION" =~ ^[A-Za-z0-9.+~:-]+$ ]] || { + echo "::error::Could not read a version from $DSC (got '$SRCPKG_VERSION')" + exit 1 + } + echo "Submitting $DSC ($SRCPKG_VERSION)" + echo "srcpkg_version=$SRCPKG_VERSION" >> "$GITHUB_OUTPUT" - name: Build in Debusine id: debusine - uses: ./.github/actions/debusine-build - with: - suite: ${{ inputs.suite }} - flavour: ${{ inputs.flavour }} - debusine-host: ${{ vars.DEBUSINE_HOST }} - debusine-scope: ${{ vars.DEBUSINE_SCOPE }} - debusine-user: ${{ secrets.DEBUSINE_USER }} - debusine-token: ${{ secrets.DEBUSINE_TOKEN }} - debusine-parent-workspace: ${{ inputs.debusine-parent-workspace || vars.DEBUSINE_PARENT_WORKSPACE || 'qli-ci' }} - extra-build-dep-workspaces: ${{ inputs.extra-build-dep-workspaces }} + env: + GITHUB_REPOSITORY_ID: ${{ github.repository_id }} + GITHUB_RUN_ID: ${{ github.run_id }} + GITHUB_RUN_ATTEMPT: ${{ github.run_attempt }} + JOB_INDEX: ${{ inputs.flavour }}-${{ inputs.suite }} + DEBUSINE_HOST: ${{ vars.DEBUSINE_HOST }} + DEBUSINE_SCOPE: ${{ vars.DEBUSINE_SCOPE }} + DEBUSINE_USER: ${{ secrets.DEBUSINE_USER }} + DEBUSINE_TOKEN: ${{ secrets.DEBUSINE_TOKEN }} + DEBUSINE_PARENT_WORKSPACE: ${{ inputs.debusine-parent-workspace || vars.DEBUSINE_PARENT_WORKSPACE || 'qli-ci' }} + SUITE: ${{ inputs.suite }} + EXTRA_BUILD_DEP_WORKSPACES: ${{ inputs.extra-build-dep-workspaces }} + run: | + # No -x here: DEBUSINE_TOKEN is in env, keep xtrace off. + set -euo pipefail + debusine-action/lib/build + # lib/build names the workspace it created and writes it out; read it + # back to report where the build can be watched. + workspace=$(sed -n 's/^workspace=//p' "$GITHUB_OUTPUT") + echo "Debusine Workspace URL: https://${DEBUSINE_HOST}/${DEBUSINE_SCOPE}/${workspace}/" \ + >> "$GITHUB_STEP_SUMMARY" # --------------------------------------------------------------------------- # publish: download the .deb files from Debusine and upload them to S3. diff --git a/README.md b/README.md index f39aea73..b8092851 100644 --- a/README.md +++ b/README.md @@ -117,13 +117,11 @@ variant is a matrix change, not a workflow redesign. | `build-kernel-debian.yml` | Builds one Debian-suite entry in Debusine, publishes it to S3, and promotes it into the workspace its caller named, if any. | Called by Daily, Release and PR build. | | `build-kernel-ubuntu.yml` | Builds one Ubuntu-suite entry on the Docker path and publishes it to S3. | Called by Daily and PR build. | -The two build workflows share their steps through two composite actions rather -than through a common orchestrator workflow: - -| Action | Used by | -| --- | --- | -| `.github/actions/prepare-kernel-source` | Both, as the `prepare` job. | -| `.github/actions/debusine-build` | The Debian workflow, as the `build` job. | +The two build workflows share their common steps through one composite action, +`.github/actions/prepare-kernel-source`, rather than through a common +orchestrator workflow. Both call it to clone the ref and build the source +package. Everything after that differs by family and is written out in the +workflow that does it. Which of them a build leg calls is decided by the caller, from the entry's suite: `resolve-matrix.py --family debian|ubuntu` splits the selection, and @@ -382,7 +380,7 @@ flowchart TD subgraph build[One build workflow per leg] C2["prepare\nprepare-kernel-source action\nClone ref, build the source package\nSkip the run if the version is published"] - C3["build\ndebusine-build action"] + C3["build\nSubmit the .dsc with lib/build"] C4["build\nprepare-kernel-source action, then\nbuild-kernel.sh --dsc in Docker"] C5["publish\nDownload .deb files, upload to S3"] C6["promote\nlib/release into the caller's workspace"] From a87d58c8c3ea5c49b9b5177e37f5606d1922f36a Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Thu, 10 Sep 2026 05:29:50 +0100 Subject: [PATCH 42/59] debian/README: correct the KVER arithmetic example The worked example of how KVER is composed added -qcom-next-20260826 to a 7.0.0-rc2 base and arrived at 7.2.0-qcom-next-20260826, so the base version on the left and the release on the right disagreed: the LOCALVERSION suffix is appended to the base version and nothing in 'prepare' rewrites it, which is the one thing the example exists to show. Say 7.2.0-rc7 on both sides. The arithmetic is the point here, not which particular release is current, and -rc7 is the version the surrounding examples already use. Signed-off-by: Christopher Obbard --- debian/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/debian/README.md b/debian/README.md index 0863f55c..897ad741 100644 --- a/debian/README.md +++ b/debian/README.md @@ -175,8 +175,8 @@ Debian's official `linux` source package and Ubuntu OEM kernels. ``` KVER = + LOCALVERSION - = 7.0.0-rc2 + -qcom-next-20260826 - = 7.2.0-qcom-next-20260826 + = 7.2.0-rc7 + -qcom-next-20260826 + = 7.2.0-rc7-qcom-next-20260826 ``` `build-kernel.sh` passes `LOCALVERSION` to `prepare`, which determines KVER by: From dcdc08c7e2443379f06a1b924fbc8a919cb955af Mon Sep 17 00:00:00 2001 From: Christopher Obbard Date: Thu, 10 Sep 2026 05:31:24 +0100 Subject: [PATCH 43/59] packaging: generate every maintainer script from a template Only the postinst was a @KVER@ template; the preinst and postrm were copied in verbatim. That left the postrm reconstructing the kernel release at run time, by stripping "linux-image-" and a trailing "-qcom" off DPKG_MAINTSCRIPT_PACKAGE. The suffix is not part of the package name any more -- the package is linux-image-, and the release carries its own +qcom-next--g -- so the strip is a no-op at best, and at worst chops a real part of the version off and runs the hooks for a kernel that does not exist. Rename all three to .in and substitute @KVER@ into each, so every script names its own kernel release the way the postinst already did. The scripts now share one shape: header, set -e, KVER, #DEBHELPER#, body. override_dh_installdeb generates them in one loop over the four maintainer script names, taking whichever templates exist, so adding a prerm becomes a matter of adding debian/linux-image.prerm.in. debian/clean and .gitignore already cover all four generated names. Which hook directories those scripts run is unchanged here, and wrong on forky; the next commit fixes that. Verified that the three generated scripts are shellcheck-clean with no @KVER@ left in them, and that the postrm now names the same release the postinst does. Signed-off-by: Christopher Obbard --- debian/README.md | 15 +++++++++++---- debian/linux-image.postinst.in | 8 +++++++- debian/linux-image.postrm | 21 --------------------- debian/linux-image.postrm.in | 26 ++++++++++++++++++++++++++ debian/linux-image.preinst | 15 --------------- debian/linux-image.preinst.in | 20 ++++++++++++++++++++ debian/rules | 20 +++++++++----------- 7 files changed, 73 insertions(+), 52 deletions(-) delete mode 100644 debian/linux-image.postrm create mode 100644 debian/linux-image.postrm.in delete mode 100644 debian/linux-image.preinst create mode 100644 debian/linux-image.preinst.in diff --git a/debian/README.md b/debian/README.md index 897ad741..4b89ed80 100644 --- a/debian/README.md +++ b/debian/README.md @@ -58,9 +58,9 @@ pkg-linux-qcom/ │ ├── changelog ← Generated by 'prepare' (gitignored) │ ├── rules ← Build logic + 'prepare' target │ ├── clean ← Lists generated files for dh_clean -│ ├── linux-image.preinst ← Pre-install maintainer script +│ ├── linux-image.preinst.in ← Pre-install maintainer script template (no-op) │ ├── linux-image.postinst.in ← Post-install maintainer script template (depmod, initramfs, GRUB) -│ ├── linux-image.postrm ← Post-remove maintainer script (GRUB update) +│ ├── linux-image.postrm.in ← Post-remove maintainer script template (GRUB update) │ ├── config/ ← Always-applied config fragments (committed) │ │ └── squashfs.config ← SQUASHFS options for Ubuntu compatibility │ ├── config-available/ ← Packaging fragments, all applied to every build @@ -89,9 +89,9 @@ pkg-linux-qcom/ | `debian/control.in` | ✅ Committed | Template with `@KVER@` placeholder | | `debian/changelog.in` | ✅ Committed | Template with `@KVER@` placeholder | | `debian/rules` | ✅ Committed | Build rules + `prepare` target | -| `debian/linux-image.preinst` | ✅ Committed | Pre-install maintainer script | +| `debian/linux-image.preinst.in` | ✅ Committed | Pre-install script template (`@KVER@` substituted at build time) | | `debian/linux-image.postinst.in` | ✅ Committed | Post-install script template (`@KVER@` substituted at build time) | -| `debian/linux-image.postrm` | ✅ Committed | Post-remove maintainer script | +| `debian/linux-image.postrm.in` | ✅ Committed | Post-remove script template (`@KVER@` substituted at build time) | | `debian/config/*.config` | ✅ Committed | Always-applied config fragments | | `debian/config-available/*.config` | ✅ Committed | Packaging fragments, all applied to every build | | `debian/dkms-modules` | 🔄 Generated | Produced by `make -f debian/rules prepare DKMS_MODULES=...` | @@ -340,6 +340,13 @@ Maintainer scripts: (triggers `update-initramfs` and `update-grub`) - `postrm` — runs `/etc/kernel/postrm.d/` hooks (triggers `update-grub`) +Each script is generated at build time from its `debian/linux-image.