Skip to content

Report the output path and sizes, and fail when no package is produced - #16

Merged
BrandonLWhite merged 7 commits into
BrandonLWhite:mainfrom
dhrimov:report-output-metadata
Aug 24, 2026
Merged

Report the output path and sizes, and fail when no package is produced#16
BrandonLWhite merged 7 commits into
BrandonLWhite:mainfrom
dhrimov:report-output-metadata

Conversation

@dhrimov

@dhrimov dhrimov commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Why

When this tool is driven from a CI script, the script needs to know which file was just written — to hash it, measure it, upload it, or hand it to Terraform. Today it cannot find out.

The packager picks the name, and the resolved path appears only inside a log line (packager.py:37). A caller that passes --output-dir is left with three bad options: re-derive distribution_name itself by parsing the TOML and copying the escaping rule from python_project.py:28; scrape stdout, which is not a documented interface; or glob the output directory, which cannot tell a fresh zip from the previous run's.

The same is true of the two size figures. Both are computed and then only logged (packager.py:62,67,69), and the uncompressed one is the number that decides whether a Lambda deploys at all.

What's here

The report — a new opt-in --report <path> flag writes a single JSON object once packaging has succeeded:

{
  "output_file": "/abs/path/lambda/my_app.zip",
  "distribution_name": "my_app",
  "output_bytes": 746972,
  "uncompressed_bytes": 2316994,
  "compressed_bytes": 746972,
  "nested_zip": false
}

A file rather than stdout, deliberately: it needs no change to how logging is configured, it cannot be corrupted by anything else the tool prints, and callers who don't pass the flag see no change at all.

One detail worth a second look. compressed_bytes is measured on the temporary dependencies zip (packager.py:67). Under the single-zip strategy that file is copied verbatim to the output, so the figure is also the size of the artifact — but under the nested strategy the outer zip is rebuilt around it, so the two differ. output_bytes is therefore a separate field, always stat() of the file at output_file, so a caller never has to branch on nested_zip to know what it's holding.

Exit non-zero when no package is produced — when the uncompressed size is over the limit and the compressed size is too, the tool printed a placeholder message, wrote nothing, and returned normally, so the process exited 0. Under set -euo pipefail that reads as success: the build goes green with no zip on disk, and whatever consumes the artifact next fails far from the cause. Now raises PackageTooLargeError, carrying both sizes and the limit.

Smaller fixes

  • re.UNICODE was being passed to re.sub as count, not flags (python_project.py:28), so it meant "replace at most 32 occurrences". Also makes the pattern a raw string — as a plain string it contains the undefined escapes \w and \d, which is a SyntaxWarning now and is slated to become an error — and removes a DeprecationWarning from the test run on 3.13.
  • The two TODO placeholder exceptions (packager.py:33, python_project.py:44) are both reached by ordinary user mistakes and now say what was looked for and where.
  • $SOURCE_DATE_EPOCH was validated per file written, so a bad value failed after the venv was built and partway through writing the zip. Now checked once at startup, and a non-integer value raises SourceDateEpochError like a pre-1980 one does, instead of a bare ValueError.
  • --output and --output-dir are now mutually exclusive, rather than --output silently winning. The group is not required, so passing neither still writes to the current directory.

Docs — the new flag and its fields, and the output filename rule stated as a contract: <output-dir>/<distribution_name>.zip, sourced from [project].name or [tool.poetry].name, case preserved. The README previously said only "with dashes replaced with underscores", which is incomplete and silent on case, so My-AppMy_App.zip was left to be discovered. That also differs from the lowercased wheel filename for the same project, which is a tempting and wrong way to predict it. The "One of the following must be specified" line is corrected too — neither was ever required.

Behaviour changes

Everything else is additive, but four things do change:

  1. The too-large case now exits non-zero instead of 0. That is the point of the fix.
  2. Passing both --output and --output-dir is now an error instead of silently preferring --output.
  3. A non-integer $SOURCE_DATE_EPOCH now raises SourceDateEpochError instead of ValueError.
  4. Project names with more than 32 runs of characters to replace are now fully normalised.

Noticed, not fixed

date_time() re-reads the environment and calls time.gmtime() for every file written, which for a large venv is tens of thousands of redundant calls. Caching it looked like an unrelated change to make here, so I left it alone.

Testing

15 new tests, in the existing style. Report contents under both strategies, the absence of a report without the flag, no report and no zip on failure, the mutually exclusive arguments, early $SOURCE_DATE_EPOCH validation leaving no artifact behind, and distribution-name normalisation including a case that fails under the old count bug.

The too-large and nested-zip report tests monkeypatch AWS_LAMBDA_MAX_UNZIP_SIZE down so small fixtures trip the thresholds — reaching the failing branch for real needs ~250 MB of incompressible data on every CI run. The existing nested test is untouched and still exercises the real constant.

Full suite passes (25 tests), and I ran it end to end against a real uv-built venv: correct path and sizes in the report, My-Smoke-AppMy_Smoke_App.zip, identical SHA-256 across two runs into the same directory, and exit 1 / exit 2 on the failure paths with no partial artifacts left behind.

Happy to split this into separate PRs if you'd rather review the fixes apart from the new flag.

Dmytro Hrimov and others added 7 commits August 24, 2026 14:35
The resolved output path and the two size figures currently exist only
inside log lines, so a script that calls this tool cannot find out which
file was just written without re-deriving the distribution name itself or
scraping stdout.

Add an opt-in `--report <path>` flag that writes a single JSON object once
packaging has succeeded:

    {
      "output_file": "/abs/path/my_app.zip",
      "distribution_name": "my_app",
      "output_bytes": 3460000,
      "uncompressed_bytes": 412000000,
      "compressed_bytes": 3456789,
      "nested_zip": false
    }

`output_bytes` is the size of the file at `output_file`, so it is correct
under both strategies. `compressed_bytes` remains the size of the
dependencies zip, which is the figure compared against the Lambda limit;
under the nested strategy the outer zip is larger, because it also holds
the loader and the stored inner zip.

Writing to a file rather than stdout keeps the report immune to anything
else the tool prints, and leaves existing output untouched for callers
that do not pass the flag.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
When the uncompressed size exceeds the Lambda limit and the compressed
size does too, the nested-zip strategy cannot help. The tool printed a
placeholder message, wrote no file, and returned normally, so the process
exited 0.

Under `set -euo pipefail` that reads as success: the build goes green with
no zip on disk, and whatever consumes the artifact next fails somewhere
far from the cause.

Raise instead, carrying both sizes and the limit so the message says how
much too large the package is. The traceback goes to stderr and the
process exits non-zero.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two ordinary user mistakes surfaced as bare tracebacks that named neither
the problem nor the fix:

  - a venv path with no `lib/python*` directory raised `Exception("input_path")`
  - a pyproject.toml with no `[project].name` or `[tool.poetry].name`
    raised `Exception("TODO Exception find_value")`

Both now say what was looked for and where, and use a fitting builtin
exception type.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`re.sub`'s fourth positional parameter is `count`, not `flags`, so
`re.sub("[^\w\d.]+", "_", self.name, re.UNICODE)` meant "replace at most 32
occurrences". Names with more than 32 runs of characters to replace were
normalised only partway.

`re.UNICODE` is already the default for str patterns in Python 3, so it can
go entirely. The pattern also becomes a raw string: as a plain string it
contains the undefined escapes `\w` and `\d`, which is a SyntaxWarning on
current Python and is slated to become an error.

Passing `count` positionally is deprecated as of 3.13, so this also removes
a DeprecationWarning from the test run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`date_time()` runs once per file written, so a bad $SOURCE_DATE_EPOCH was
only noticed after the venv had been built and while the zip was being
produced. Call it once at startup so the failure lands before any work.

A non-integer value also gave a bare ValueError traceback while a pre-1980
value gave SourceDateEpochError. Both now raise SourceDateEpochError, and
the message quotes the offending value.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Passing both was accepted, with --output silently winning. Put them in an
argparse mutually exclusive group so the conflict is reported instead.

The group is not required: --output-dir keeps its default of the current
working directory, so invocations that pass neither are unaffected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Document the new --report flag and each field of the JSON it writes.

State the output filename rule as a contract: `<output-dir>/<distribution_name>.zip`,
where the name comes from `[project].name` or `[tool.poetry].name`, and
case is preserved. The README previously said only "with dashes replaced
with underscores", which is both incomplete (every run of characters
outside `A-Za-z0-9_.` is replaced) and silent on case, so `My-App` yielding
`My_App.zip` was left for callers to discover. That also differs from the
lowercased wheel filename for the same project, which is a tempting and
wrong way to predict it.

Also correct "One of the following must be specified" for `--output` /
`--output-dir`: neither is required, and they cannot now be combined.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@BrandonLWhite
BrandonLWhite merged commit cae8e93 into BrandonLWhite:main Aug 24, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants