Grype Scan CI job fails: docker image tag missing PR_NUMBER prefix due to boto3 1.43.35 / botocore 1.34.146 runtime incompatibility

[CarlosFelipeOR]

Type of problem

Bug report — CI/CD infrastructure (Grype Scan workflow).

Describe the situation

Starting around 2026-06-22, the Grype Scan job has been failing on multiple branches (antalya-26.5, antalya-26.3, stable-26.3). The job fails at the docker pull step with:

DOCKER_IMAGE=altinityinfra/clickhouse-server:-26.3.13.10001.altinitytest
++ docker pull altinityinfra/clickhouse-server:-26.3.13.10001.altinitytest
invalid reference format
Error: Process completed with exit code 1.

Note the malformed tag :-26.3.13.10001.altinitytest — the PR_NUMBER prefix (expected 0 for non-PR builds) is missing, so the tag starts with :- instead of :0-. The actual valid tag on Docker Hub is 0-26.3.13.10001.altinitytest.

Failing runs

• https://github.com/Altinity/ClickHouse/actions/runs/27984880626/job/82838587623
• https://github.com/Altinity/ClickHouse/actions/runs/27902119388/job/82570661678
• https://github.com/Altinity/ClickHouse/actions/runs/27848500950/job/82430378922

Last known passing run

• https://github.com/Altinity/ClickHouse/actions/runs/27790496205/job/82249103885

Root cause

The Grype Scan workflow (.github/workflows/grype_scan.yml) builds the image tag from the output of tests/ci/version_helper.py:

python3 ./tests/ci/version_helper.py | grep = | tee /tmp/version_info
source /tmp/version_info
...
echo "docker_image=${{ inputs.docker_image }}:$PR_NUMBER-$VERSION$TAG_SUFFIX" >> $GITHUB_OUTPUT

version_helper.py only prints the PR_NUMBER= line if it can import PRInfo from pr_info.py:

try:
    # grype scan needs to know the PR number
    # But non-grype jobs might be missing dependencies
    from pr_info import PRInfo
except ImportError:
    PRInfo = None
...
if PRInfo:
    pr_info = PRInfo()
    print(f"PR_NUMBER={pr_info.number}")

pr_info.py transitively imports boto3 (via get_robot_token.py). The Set up Python step in the workflow runs:

pip install --upgrade requests chardet urllib3 unidiff boto3 PyGithub
pip install testflows==$TESTFLOWS_VERSION awscli==1.33.28

Without a version pin, pip picks the latest boto3. Then awscli==1.33.28 forces botocore==1.34.146, leaving an incompatible combination at runtime.

Pip output from the last passing run (4 days ago)

Successfully installed PyGithub-2.9.1 boto3-1.43.33 botocore-1.43.33 ... s3transfer-0.19.0 ...

Pip output from a failing run

Successfully installed PyGithub-2.9.1 boto3-1.43.35 botocore-1.43.35 ... s3transfer-0.19.0 ...

⚠️ Note: the pip dependency-resolver warning

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed.
boto3 X.Y.Z requires botocore<1.44.0,>=X.Y.Z, but you have botocore 1.34.146 which is incompatible.
boto3 X.Y.Z requires s3transfer<0.20.0,>=0.19.0, but you have s3transfer 0.10.4 which is incompatible.

is not the causal signal. It is also present in the last passing run (e.g. boto3 1.43.33 vs botocore 1.34.146, see https://github.com/Altinity/ClickHouse/actions/runs/27826575495/job/82367089469). The real symptom is a runtime ImportError (see below) that only started occurring with boto3 >= 1.43.35.

Real cause: silently swallowed runtime ImportError

Between boto3 1.43.33 and boto3 1.43.35, boto3.docs.utils started importing DocumentModifiedShape from botocore.docs.utils, a symbol that does not exist in botocore 1.34.146. As a result, import boto3 now raises:

ImportError: cannot import name 'DocumentModifiedShape' from 'botocore.docs.utils'

This traceback does not appear in the CI log, because the try/except ImportError block in version_helper.py (shown above) swallows it silently. So in the CI log the only visible symptom is the absence of the PR_NUMBER= line in the output of version_helper.py:

CLICKHOUSE_VERSION_REVISION='54520'
CLICKHOUSE_VERSION_MAJOR='26'
...
CLICKHOUSE_VERSION_FLAVOUR='altinityantalya'   ← no `PR_NUMBER=` line emitted

→ PRInfo = None → $PR_NUMBER is empty in the shell after source /tmp/version_info → tag becomes :-26.3.13.10001.altinitytest.

The traceback above was obtained by reproducing the workflow's pip install steps locally (not from the CI log itself).

The failure is not caused by any PR change — it was triggered purely by the upstream boto3 1.43.35 release combined with the workflow's --upgrade (unpinned) install.

Locally reproduced

Scenario	boto3	botocore (after awscli==1.33.28)	pip resolver warning	import boto3 works?	version_helper.py prints PR_NUMBER=?
Last passing run	1.43.33	1.34.146	yes	yes	yes
Failing run	1.43.35 / 1.43.36	1.34.146	yes	no (ImportError: DocumentModifiedShape)	no

How to reproduce locally

python3 -m venv /tmp/v && source /tmp/v/bin/activate
pip install --upgrade requests chardet urllib3 unidiff boto3 PyGithub
pip install awscli==1.33.28
cd tests/ci && python3 -c "from pr_info import PRInfo"
# ImportError: cannot import name 'DocumentModifiedShape' from 'botocore.docs.utils'

Pinning boto3==1.43.33 in the first step makes the import succeed and version_helper.py then prints PR_NUMBER=0 as expected.

Expected behavior

Grype Scan produces a well-formed image tag (:0-<version>...) and the workflow completes.

Proposed fix

Minimal/short-term fix (this issue): pin boto3 to the last known good version in .github/workflows/grype_scan.yml:

-            pip install --upgrade requests chardet urllib3 unidiff boto3 PyGithub
+            pip install --upgrade requests chardet urllib3 unidiff 'boto3==1.43.33' PyGithub

Follow-up (separate work):

• Upgrade awscli to a version that uses botocore>=1.43 so boto3 no longer needs to be pinned.
• Make version_helper.py surface a warning (instead of swallowing the ImportError silently) so this kind of dependency breakage is visible immediately rather than producing a malformed tag. Without this, the symptom of any future breakage of this import path will remain invisible in CI logs, as it was here.

Affected branches

• antalya-26.5
• antalya-26.3
• stable-26.3