Skip to content

Add structured artifact-contract diff reporting #85

Description

@stacknil

The current release gate can tell whether committed artifacts match regenerated output, but the failure mode is mostly binary: either the artifacts match or they do not. A future standalone technical track could add a structured artifact-contract diff report so a maintainer can quickly see what changed before deciding whether to accept regenerated outputs.

This is not urgent for the current v1.x contract, but it is a clean independent project because it touches comparison logic, schemas, fixtures, and tests without adding another demo.

Potential scope:

  • Compare two artifact trees, for example committed artifacts vs a regenerated output directory.
  • Report missing files, extra files, changed files, schema-version changes, and run-manifest digest changes.
  • For JSON/JSONL artifacts, produce lightweight structural summaries such as record counts, top-level keys, and schema IDs/versions where available.
  • Emit both a human-readable summary and a machine-readable JSON report.
  • Keep text/Markdown report diffs shallow; the goal is contract triage, not a full prose diff engine.

Possible CLI shape:

python scripts/artifact_contract_diff.py --expected . --actual /tmp/telemetry-lab-regenerated --json-out artifact_diff.json

Acceptance criteria for a future PR:

  • Add fixture-based tests for unchanged artifacts, missing/extra files, changed JSON records, changed schema versions, and changed run-manifest digests.
  • Keep comparison deterministic across platforms, including line-ending normalization for text artifacts.
  • Do not replace python scripts/regenerate_artifacts.py --check; this should complement the existing gate by explaining mismatches.
  • Document the output fields in a short technical note only if the JSON report becomes reviewer-facing.

Boundaries:

  • No new demo.
  • No live telemetry or account integration.
  • No dashboard, alert routing, case management, autonomous response, or incident verdict.
  • No production SIEM claim.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesttestingTest coverage and edge-case behavior

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions