Skip to content

Stop reporting a failed comparison as a clean result - #199

Merged
reuvenharrison merged 2 commits into
mainfrom
fix/comparison-failure-not-reported-as-clean
Aug 10, 2026
Merged

Stop reporting a failed comparison as a clean result#199
reuvenharrison merged 2 commits into
mainfrom
fix/comparison-failure-not-reported-as-clean

Conversation

@reuvenharrison

@reuvenharrison reuvenharrison commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

When oasdiff cannot load or parse a spec (exit code >= 2), the breaking and diff actions printed the error annotation and then carried straight on to write No breaking changes / No changes.

The step still failed, so CI caught it, but everything a person actually reads described the run as clean: the step output, the job summary, and the PR comment. It is also wrong for anyone using continue-on-error.

The fix

Exit as soon as the comparison fails. This is what the changelog action already does:

if [ "$exit_code" -ge 2 ]; then
    [ -s "$_err" ] && echo "::error::$(tr '\n' ' ' < "$_err")"
    if [ "$exit_code" -eq 123 ]; then
        echo "::error::…allow-external-refs remedy…"
    fi
    rm -f "$_err"
    exit "$exit_code"
fi

breaking and diff were simply missing the early exit. Nothing downstream has anything to report about a comparison that never ran, and stopping there keeps the existing 123 remedy message as the most specific thing the user sees rather than burying it under a generic failure notice.

validate had it too

Reviewing the other two actions turned up the same defect in validate, so it is fixed here rather than left for a follow-up. It shows up in the outputs instead of the text, which is worse: after the annotation it continued to a second oasdiff validate run that fails identically and prints nothing, and the count reading that output cannot distinguish it from a valid spec.

# A valid spec prints nothing, so the count stays 0.

An unloadable spec therefore published findings=0, error_count=0, warning_count=0, info_count=0. The step failed, but anything gating on those outputs reads a broken spec as a clean one, and unlike the text case that is machine-readable.

changelog is the only one of the four that was already correct.

Tests

One regression job per action, using a spec whose $ref cannot be resolved (exit 102). Each asserts the step fails and that the output never contains the clean-result text, so a future refactor cannot reintroduce the fallthrough. The validate job asserts the finding counts are absent rather than zero.

Worth noting the existing test-error-annotation.yaml already loops over all five actions at exit 102, validate included. It passed throughout, because it only checks that an annotation is emitted, which validate always did before carrying on. That is why the defect survived.

Credit

Found and reported by @lucas-monteiro-g in #198, along with the observation that the failure has to stay observable in a fixture. That PR also added comparison-status and comparison-error outputs; those are left out here since the step's own failure already signals the condition and new outputs are permanent API.

When oasdiff cannot load or parse a spec (exit code >= 2), the breaking and
diff actions printed the error annotation and then carried on to write
'No breaking changes' / 'No changes'. The step still failed, so CI caught
it, but the output, job summary and PR comment all described the run as
clean, which is misleading for anyone reading them or using
continue-on-error.

Exit as soon as the comparison fails, which is what the changelog action
already does. Nothing downstream has anything to report about a comparison
that never ran.

Adds a regression test per action using a spec with an unresolvable $ref,
asserting the step fails and never emits the clean-result text.

Reported by @lucas-monteiro-g in #198.
validate had the same fallthrough as breaking and diff, but it shows up
in the outputs rather than in the text. After printing the annotation it
carried on to Run 2, which fails the same way and prints nothing, and
the count that reads it cannot tell that from a valid spec:

    # A valid spec prints nothing, so the count stays 0.

So an unloadable spec published findings=0, error_count=0,
warning_count=0, info_count=0. The step still failed, but a workflow
gating on those outputs reads a broken spec as a clean one, which is
worse than the text case because it is machine-readable.

Same early exit as the other three. The existing error-annotation test
loops over validate already, but only asserts the annotation is emitted,
which it always was; the new job asserts the counts are absent.
@reuvenharrison
reuvenharrison merged commit 5ad1d21 into main Aug 10, 2026
36 checks passed
@reuvenharrison
reuvenharrison deleted the fix/comparison-failure-not-reported-as-clean branch August 10, 2026 20:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant