Skip to content

Stop pretending include-checks works, and point it at .oasdiff.yaml - #197

Merged
reuvenharrison merged 1 commit into
mainfrom
fix/severity-levels-input
Aug 10, 2026
Merged

Stop pretending include-checks works, and point it at .oasdiff.yaml#197
reuvenharrison merged 1 commit into
mainfrom
fix/severity-levels-input

Conversation

@reuvenharrison

@reuvenharrison reuvenharrison commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Closes #196.

The bug

oasdiff retired the optional-checks mechanism include-checks drove, and now ignores the flag:

"include-checks": "it is now ignored; use --severity-levels to enforce a check as breaking"

The actions still advertised it as functional. So the documented way to enforce a check did nothing at all: the user believes a check is enforced, no error appears, and it never fires. Silence is the worst of the available failure modes, and it is the whole of the bug.

The fix

The action no longer forwards the flag, and emits a warning annotation instead:

::warning::oasdiff: 'include-checks' is ignored, the optional-checks mechanism it drove was
retired. To make a check fail the build, add 'severity-levels: <file>' to .oasdiff.yaml,
listing the check id with level 'err', and set 'fail-on: ERR'.

The input itself stays, so workflows that pass it today keep working rather than failing on an unknown input. Forwarding it was pointless (oasdiff ignores it) and would break outright if the flag is ever removed rather than merely ignored.

Why there is no severity-levels input

#196 proposed adding one. It is deliberately not here.

.oasdiff.yaml is the configuration surface these actions converge on, and the inputs are a deliberate subset, not an aspiration to cover the flag set. Counting breaking:

count examples
oasdiff flags with an input 10 fail-on, composed, err-ignore
oasdiff flags without 20 auto-upgrade, stability-level, match-path, prefix-base, format, lang, template

So not having an input is the norm. severity-levels already worked through the config file before this PR, needing no code at all, and adding an input would grow a surface we intend to stop growing. It would also add another positional argument, in a scheme where a mis-ordered index corrupts a run silently rather than erroring.

The deprecation message and the README therefore teach the config file:

# .oasdiff.yaml
severity-levels: ./oasdiff-levels.txt
fail-on: ERR
# oasdiff-levels.txt
api-version-not-bumped          err
api-major-version-not-bumped    err

That is the recipe #154 needed.

Tests

  • severity-levels via .oasdiff.yaml — downgrades api-removed-without-deprecation (this repo's only breaking finding on the test specs) to info, and asserts fail-on: ERR stops failing. Sits alongside the existing yaml-config jobs for fail-on and err-ignore, so the config path is pinned rather than assumed.
  • include-checks — passes not-a-real-check and asserts the run is unaffected. This fails if the action ever forwards the input again, since oasdiff would reject the name.

Also verified by running the entrypoint against a stubbed oasdiff that logs its argv: neither --include-checks nor --severity-levels reaches the CLI, and the annotation is emitted.

Closes #196.

oasdiff retired the optional-checks mechanism include-checks drove and
now ignores the flag. The actions still advertised it as functional, so
the documented way to enforce a check silently did nothing: the user
believes a check is enforced, no error appears, and it never fires. That
is the worst of the available failure modes, and it is the bug here.

The action no longer forwards the flag and emits a warning annotation
instead. Forwarding it was pointless, since oasdiff ignores it, and
would break outright if the flag is ever removed rather than ignored.
The input stays so workflows that set it today keep working.

The replacement, severity-levels, is deliberately NOT added as an input.
.oasdiff.yaml is the configuration surface these actions converge on,
and the inputs are a deliberate subset: 20 of oasdiff's 30 breaking
flags have no input, including auto-upgrade, stability-level, match-path
and format. Adding one more would grow a surface we intend to stop
growing, and every input is another positional argument in a scheme
where a mis-ordered index corrupts a run silently. The deprecation
message and the README therefore teach the config file, which already
worked and needed no code change.

Tests: one job configures severity-levels through .oasdiff.yaml and
asserts the downgrade takes effect, alongside the existing yaml-config
jobs for fail-on and err-ignore. Another passes a check name that does
not exist to include-checks and asserts the run is unaffected, which
fails if the action ever forwards it again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@reuvenharrison
reuvenharrison force-pushed the fix/severity-levels-input branch from 4ee5d59 to 4c04ab4 Compare August 9, 2026 22:33
@reuvenharrison reuvenharrison changed the title Add a severity-levels input, and stop pretending include-checks works Stop pretending include-checks works, and point it at .oasdiff.yaml Aug 9, 2026
@reuvenharrison
reuvenharrison merged commit aac8c66 into main Aug 10, 2026
33 checks passed
@reuvenharrison
reuvenharrison deleted the fix/severity-levels-input branch August 10, 2026 07:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

include-checks is advertised but ignored; severity-levels has no input

1 participant