Skip to content

Add --xfail-flaky to report exhausted flaky failures as xfail - #378

Open
LouisDeconinck wants to merge 2 commits into
pytest-dev:masterfrom
LouisDeconinck:issue-92-xfail-flaky
Open

LouisDeconinck wants to merge 2 commits into
pytest-dev:masterfrom
LouisDeconinck:issue-92-xfail-flaky

Conversation

@LouisDeconinck

@LouisDeconinck LouisDeconinck commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Adds an opt-in --xfail-flaky flag: a test that still fails after all its reruns are exhausted is reported as xfailed instead of failed (functionality similar to pytest-ignore-flaky, as requested)
  • Only applies to tests that were actually rerun (execution_count > 1); tests failing on first attempt or excluded by --only-rerun/--rerun-except still report as failed
  • Implemented by converting the final call report to skipped + wasxfail, so pytest's standard xfail reporting applies

Closes #92

Test plan

  • Always-failing test with --reruns 2 --xfail-flaky → 1 xfailed, 2 rerun, exit code 0
  • Eventual pass unaffected; tests without reruns unaffected; marker-based reruns covered
  • --only-rerun non-matching error still reports failed

A test that still fails after all its reruns are exhausted is reported as
xfailed instead of failed, similar to pytest-ignore-flaky. Only tests that
actually reran are affected, and failures excluded by
--only-rerun/--rerun-except still fail.

Closes pytest-dev#92.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

@sylvesterkaczmarek sylvesterkaczmarek left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--xfail-flaky only converts the final report when report.when == 'call'. Reruns can also be triggered by setup or teardown failures, so a fixture that fails on every attempt can exhaust its reruns and still finish as a normal failure even though the option says exhausted flaky failures are reported as xfailed. Could the terminal setup/teardown failure be handled too, or the option be explicitly narrowed, with a setup-failure regression test?

The conversion only applied to call-phase reports, so a fixture that
failed on every attempt still finished as a normal failure.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@LouisDeconinck

Copy link
Copy Markdown
Contributor Author

Fixed in 53b61ec: the xfail conversion now applies to the terminal failed report in any phase, so a fixture that fails on every attempt ends as xfailed. Added a setup-failure regression test.

@LouisDeconinck

Copy link
Copy Markdown
Contributor Author

Fixed — exhausted setup/teardown failures are now converted to xfail too, not just call reports, with a setup-failure regression test (53b61ec).

sylvesterkaczmarek

This comment was marked as duplicate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Can we add functionality similar to pytest-ignore-flaky plugin

2 participants