Skip to content

feat(source-files): wire archive overlays into prep-sources with post-overlay hash tracking#276

Open
Tonisal-byte wants to merge 5 commits into
mainfrom
asalinas/tarball-overlays-2
Open

feat(source-files): wire archive overlays into prep-sources with post-overlay hash tracking#276
Tonisal-byte wants to merge 5 commits into
mainfrom
asalinas/tarball-overlays-2

Conversation

@Tonisal-byte

@Tonisal-byte Tonisal-byte commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Wires archive-scoped overlays into the source preparation pipeline.

This applies archive overlays during prep-sources, repacks modified archives, refreshes their sources file hashes, and adds origin.type = "overlay" for recording expected post-overlay archive hashes. It also ensures overlay-origin source files are not downloaded separately and validates stale configured hashes during full source prep.

Example:

# Declares the expected post-overlay archive hash. This must have the same
# filename as the archive targeted below and replaces its upstream `sources`
# entry during render.
[[components.example.source-files]]
filename = "example-1.0.tar.gz"
hash = "..."
hash-type = "SHA2512"
origin = { type = "overlay" }
replace-upstream = true
replace-reason = "Remove bundled vendor files not shipped by Azure Linux"

# Multiple archive overlays can use the same overlay-origin entry.
[[components.example.overlays]]
type = "file-remove"
archive = "example-1.0.tar.gz"
file = "vendor/**"
description = "Remove bundled dependencies"

@Tonisal-byte
Tonisal-byte force-pushed the asalinas/tarball-overlays-1 branch from c8b6390 to 0a209cc Compare July 14, 2026 17:19
@Tonisal-byte
Tonisal-byte force-pushed the asalinas/tarball-overlays-2 branch from bae831d to 8635511 Compare July 15, 2026 21:25
@Tonisal-byte
Tonisal-byte changed the base branch from asalinas/tarball-overlays-1 to main July 15, 2026 21:50
Copilot AI review requested due to automatic review settings July 15, 2026 23:12
@Tonisal-byte
Tonisal-byte marked this pull request as ready for review July 15, 2026 23:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR wires archive-scoped overlays into the prep-sources pipeline: archives modified by overlays are deterministically repacked, their corresponding sources entries are re-hashed, and configuration can record/validate expected post-overlay hashes via a new origin.type = "overlay" source-file origin.

Changes:

  • Add new OriginTypeOverlay / schema enum value "overlay" and associated config validation rules for overlay-origin source-files entries.
  • Apply archive overlays during prep-sources, repack modified archives, and rehash only the affected sources entries; additionally validate configured post-overlay hashes during full prep.
  • Adjust Fedora lookaside extraction to avoid skipping files that are not actually fetched by FetchFiles (e.g., overlay-origin entries).

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
schemas/azldev.schema.json Adds "overlay" to the origin.type enum in the JSON schema.
scenario/snapshots/TestSnapshotsContainer_config_generate-schema_stdout_1.snap Updates schema snapshot output for the new enum value.
scenario/snapshots/TestSnapshots_config_generate-schema_stdout_1.snap Updates schema snapshot output for the new enum value.
internal/providers/sourceproviders/sourcemanager.go Skips downloads for overlay-origin refs; refactors provider attempts into helper; adds internal guard for overlay origin in download path.
internal/providers/sourceproviders/fedorasourceprovider.go Avoids skipping upstream lookaside downloads for non-fetched origin types (e.g., overlay).
internal/projectconfig/fingerprint_test.go Improves fingerprint tag parsing/validation to account for hashstructure tag options.
internal/projectconfig/configfile.go Adds overlay-origin validation rules and calls archive-overlay/origin validation during config validation.
internal/projectconfig/configfile_test.go Adds validation coverage for archive overlay requiring an overlay-origin entry.
internal/projectconfig/component.go Introduces OriginTypeOverlay, OriginType.IsFetched, and ValidateArchiveOverlayOrigins.
internal/app/azldev/core/sources/sourceprep.go Applies archive overlays first, tracks repacked archives, rehashes corresponding sources entries, and validates configured post-overlay hashes.
internal/app/azldev/core/sources/sourceprep_test.go Adds end-to-end tests around archive overlay repacking + sources rehashing behavior.
internal/app/azldev/core/sources/archiveoverlays.go Implements batching of archive overlays per archive (single extract/modify/repack cycle per archive).
internal/app/azldev/core/sources/archiveoverlays_test.go Adds coverage for archive overlay glob semantics and end-to-end behavior via PrepareSources.
internal/app/azldev/core/sources/archiveoverlays_internal_test.go Adds direct unit coverage for archive overlay grouping behavior.
internal/app/azldev/core/components/resolver.go Validates archive overlay/origin invariants after config resolution/overlay-file expansion.
docs/user/reference/config/overlays.md Documents drift-protection requirement for archive overlays via overlay-origin source-files.
docs/user/reference/config/components.md Documents the new "overlay" origin type and the workflow for recording post-overlay hashes.

Comment thread internal/providers/sourceproviders/fedorasourceprovider.go Outdated
Comment thread internal/projectconfig/configfile.go Outdated
Copilot AI review requested due to automatic review settings July 16, 2026 18:10
Antonio Salinas added 4 commits July 16, 2026 18:10
Add the archive extract/apply/repack engine plus the ComponentOverlay 'archive' field, validation, and fingerprint handling. Extends internal/utils/archive with compression sniffing, ExtractAuto, and strict unsupported-entry handling. Pipeline wiring and hash tracking follow in a subsequent change.
…-overlay hash tracking

Apply archive overlays in the source-prep pipeline, rehash repacked archives in the 'sources' file, and add the 'overlay' source-file origin type that records and validates the expected post-overlay hash.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 4 comments.

Comment thread internal/projectconfig/configfile.go Outdated
Comment thread internal/projectconfig/configfile.go Outdated
Comment thread internal/projectconfig/configfile.go
Comment thread internal/projectconfig/component.go
Copilot AI review requested due to automatic review settings July 16, 2026 18:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.

Comment thread internal/app/azldev/core/sources/sourceprep.go
Comment thread internal/projectconfig/configfile.go
Comment thread internal/projectconfig/component.go
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants