Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -712,7 +712,8 @@ jobs:
"Accessibility=suite-artifacts/accessibility.json" \
"Visual Regression=suite-artifacts/reports/visual.json" \
"Performance=suite-artifacts/performance.json" \
"Load=suite-artifacts/load.json"
"Load=suite-artifacts/load.json" \
|| echo "TESTS_FAILED=1" >> "$GITHUB_ENV"
echo ""
echo "**Platform:** ${{ env.BASE_URL }}"
echo ""
Expand All @@ -739,3 +740,11 @@ jobs:
reports/e2e_combined.json
reports/e2e_summary.md
retention-days: 30

# Every suite's pytest step is `continue-on-error` so all reports get
# produced; this is where a failing test finally turns the run red.
- name: Fail on test failures
if: env.TESTS_FAILED == '1'
run: |
echo "::error::One or more suites have failing tests — see the results table in the job summary."
exit 1
8 changes: 4 additions & 4 deletions scripts/suite_summary.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@

Each argument is `Label=path`. Missing or unparseable files are reported as
such rather than skipped silently — a suite whose report never arrived is a
result worth seeing, not a blank row. Shard globs are supported so the E2E
result worth seeing, not a blank row. Exits 1 when any suite has failing
tests, so CI can gate on it. Shard globs are supported so the E2E
matrix can be passed as one label:

python scripts/suite_summary.py "E2E=reports/e2e_shard_*.json"
Expand Down Expand Up @@ -112,9 +113,8 @@ def main(argv: list[str]) -> int:
print("\n".join(rows))
print()
if any_failure:
print("> One or more suites have failing tests. Job status alone does not")
print("> reflect this — every suite runs with `continue-on-error: true`.")
return 0
print("> One or more suites have failing tests.")
return 1 if any_failure else 0


if __name__ == "__main__":
Expand Down
Loading