Stop reporting a failed comparison as a clean result - #199
Merged
reuvenharrison merged 2 commits intoAug 10, 2026
Conversation
When oasdiff cannot load or parse a spec (exit code >= 2), the breaking and diff actions printed the error annotation and then carried on to write 'No breaking changes' / 'No changes'. The step still failed, so CI caught it, but the output, job summary and PR comment all described the run as clean, which is misleading for anyone reading them or using continue-on-error. Exit as soon as the comparison fails, which is what the changelog action already does. Nothing downstream has anything to report about a comparison that never ran. Adds a regression test per action using a spec with an unresolvable $ref, asserting the step fails and never emits the clean-result text. Reported by @lucas-monteiro-g in #198.
validate had the same fallthrough as breaking and diff, but it shows up
in the outputs rather than in the text. After printing the annotation it
carried on to Run 2, which fails the same way and prints nothing, and
the count that reads it cannot tell that from a valid spec:
# A valid spec prints nothing, so the count stays 0.
So an unloadable spec published findings=0, error_count=0,
warning_count=0, info_count=0. The step still failed, but a workflow
gating on those outputs reads a broken spec as a clean one, which is
worse than the text case because it is machine-readable.
Same early exit as the other three. The existing error-annotation test
loops over validate already, but only asserts the annotation is emitted,
which it always was; the new job asserts the counts are absent.
reuvenharrison
deleted the
fix/comparison-failure-not-reported-as-clean
branch
August 10, 2026 20:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When oasdiff cannot load or parse a spec (exit code >= 2), the
breakinganddiffactions printed the error annotation and then carried straight on to writeNo breaking changes/No changes.The step still failed, so CI caught it, but everything a person actually reads described the run as clean: the step output, the job summary, and the PR comment. It is also wrong for anyone using
continue-on-error.The fix
Exit as soon as the comparison fails. This is what the
changelogaction already does:breakinganddiffwere simply missing the early exit. Nothing downstream has anything to report about a comparison that never ran, and stopping there keeps the existing 123 remedy message as the most specific thing the user sees rather than burying it under a generic failure notice.validatehad it tooReviewing the other two actions turned up the same defect in
validate, so it is fixed here rather than left for a follow-up. It shows up in the outputs instead of the text, which is worse: after the annotation it continued to a secondoasdiff validaterun that fails identically and prints nothing, and the count reading that output cannot distinguish it from a valid spec.An unloadable spec therefore published
findings=0,error_count=0,warning_count=0,info_count=0. The step failed, but anything gating on those outputs reads a broken spec as a clean one, and unlike the text case that is machine-readable.changelogis the only one of the four that was already correct.Tests
One regression job per action, using a spec whose
$refcannot be resolved (exit 102). Each asserts the step fails and that the output never contains the clean-result text, so a future refactor cannot reintroduce the fallthrough. Thevalidatejob asserts the finding counts are absent rather than zero.Worth noting the existing
test-error-annotation.yamlalready loops over all five actions at exit 102,validateincluded. It passed throughout, because it only checks that an annotation is emitted, whichvalidatealways did before carrying on. That is why the defect survived.Credit
Found and reported by @lucas-monteiro-g in #198, along with the observation that the failure has to stay observable in a fixture. That PR also added
comparison-statusandcomparison-erroroutputs; those are left out here since the step's own failure already signals the condition and new outputs are permanent API.