Skip to content

ci(): check PYTHON_VERSIONS against DuckDB wheels and Lambda runtimes #29

Merged
bengeois merged 1 commit into
bengeois:mainfrom
Gudsfile:feat/check-python-versions
Jun 30, 2026
Merged

ci(): check PYTHON_VERSIONS against DuckDB wheels and Lambda runtimes #29
bengeois merged 1 commit into
bengeois:mainfrom
Gudsfile:feat/check-python-versions

Conversation

@Gudsfile

@Gudsfile Gudsfile commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator

Follows up on #25.

Adds a pull_request workflow that fails if PYTHON_VERSIONS in build-layer.yml diverges from the intersection of DuckDB manylinux wheels (from PyPI) and supported Lambda runtimes (hardcoded).

Why

The build matrix only validates versions that are already listed, it cannot detect when DuckDB ships wheels for a new Python version absent from the matrix. This check fills that gap, and also catches stale versions (DuckDB dropped support, or Lambda runtime was removed).

Allows you to safely use auto-merge without requiring a review (#28).

What does it look like

python3 scripts/check_python_versions.py
PYTHON_VERSIONS is correct for DuckDB 1.5.4.

####################################

PYTHON_VERSIONS mismatch for DuckDB 1.4.2:
  + 3.9  (add to PYTHON_VERSIONS)

####################################

PYTHON_VERSIONS mismatch for DuckDB 1.5.2:
  - 3.15  (not in DuckDB wheels and not in Lambda runtimes, remove from PYTHON_VERSIONS)

####################################

Error: DuckDB 1.5.5 not found on PyPI.

Tests

To avoid making noise on this PR, I tested it in another PR: #31

Open question

  1. This is a standalone workflow rather than a job in build-layer.yml. Open to merging it into build-layer.yml on pull_request events if preferred.
  2. Should I create unit tests for check_python_versions.py and a CI to run them?

Out of scope

  • Lambda runtimes update: update manually when AWS Lambda adds a new runtime.
    A follow-up could automate this by scrapping https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html (or any better idea).
  • PYTHON_VERSIONS matrix:
    update manually when DuckDB adds/drops Python support or when AWS Lambda adds a new runtime.
    A follow-up could automate this by computing the intersection of DuckDB wheels (from PyPI metadata) and supported Lambda runtimes (hardcoded, changes rarely) and updating the matrix directly in the bump PR.
    originally posted in feat(): automate DUCKDB_VERSION bumps #25 (comment)

@Gudsfile Gudsfile self-assigned this Jun 18, 2026
@Gudsfile Gudsfile force-pushed the feat/check-python-versions branch 4 times, most recently from f708077 to b031f67 Compare June 21, 2026 15:44
@Gudsfile Gudsfile marked this pull request as ready for review June 21, 2026 15:53
@Gudsfile Gudsfile requested a review from bengeois as a code owner June 21, 2026 15:53

@bengeois bengeois left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Gudsfile 🙏🏼

Open questions

  1. I'd keep it standalone. The triggers are different (no need to run on workflow_dispatch) and it keeps build-layer.yml focused on the build. No strong reason to merge them.
  2. Not critical given the script's size, but it would be worth adding tests for fetch_duckdb_pythons (with a mocked HTTP response) and read_python_versions (with a fixture YAML) to catch silent regressions if the PyPI response format ever changes.

Suggestion

  1. Also, on the bump PRs created by #25, since the branch belongs to the repo (not a fork), the bump-duckdb.yml workflow could theoretically compute the correct PYTHON_VERSIONS at bump time, by running the same intersection logic and write it directly into the bump commit, rather than failing the check after the fact. This would make the whole pipeline fully hands-off.
    Is that something you'd consider tackling, either in this PR or as a follow-up?

@Gudsfile Gudsfile force-pushed the feat/check-python-versions branch from b031f67 to fe2f611 Compare June 23, 2026 15:20
@Gudsfile

Copy link
Copy Markdown
Collaborator Author

Standalone workflow
Agreed.

Tests
Added in this PR:

  • TestFetchDuckdbPythons: mocks urllib.request.urlopen with a realistic PyPI response to cover manylinux extraction, non-manylinux filtering, 404, and HTTP errors
  • TestReadPythonVersions: uses tmp_path + patch.object(repo_root) as a fixture to cover happy path, whitespace stripping, and missing key

Both wired into CI in check-python-versions.yml.

Suggestion (bump-time computation)
Good idea, this is tracked in a separate draft PR (#30) and will be addressed as a follow-up.

Comment thread scripts/check_python_versions.py Outdated
Add a pull_request workflow that fails if PYTHON_VERSIONS in build-layer.yml
diverges from the intersection of DuckDB manylinux wheels (from PyPI) and
supported Lambda runtimes (.github/.lambda-python-runtimes).

Catches both missing versions (DuckDB added a Python not in the matrix) and
stale versions (DuckDB dropped a Python still in the matrix or a Lambda runtime
was removed).

Lambda runtimes are hardcoded in .github/.lambda-python-runtimes and
must be updated manually when AWS announces a new runtime.

Print an error when the DuckDB version in .duckdb-version does not exist on
PyPI instead of crashing with an unhandled HTTPError.
@Gudsfile Gudsfile force-pushed the feat/check-python-versions branch from fe2f611 to 561f1c6 Compare June 29, 2026 22:10
@Gudsfile Gudsfile requested a review from bengeois June 29, 2026 22:12

@bengeois bengeois left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Gudsfile 🙏

@bengeois bengeois merged commit bfb961b into bengeois:main Jun 30, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants