Add --xfail-flaky to report exhausted flaky failures as xfail - #378
LouisDeconinck wants to merge 2 commits into
Conversation
A test that still fails after all its reruns are exhausted is reported as xfailed instead of failed, similar to pytest-ignore-flaky. Only tests that actually reran are affected, and failures excluded by --only-rerun/--rerun-except still fail. Closes pytest-dev#92. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
sylvesterkaczmarek
left a comment
There was a problem hiding this comment.
--xfail-flaky only converts the final report when report.when == 'call'. Reruns can also be triggered by setup or teardown failures, so a fixture that fails on every attempt can exhaust its reruns and still finish as a normal failure even though the option says exhausted flaky failures are reported as xfailed. Could the terminal setup/teardown failure be handled too, or the option be explicitly narrowed, with a setup-failure regression test?
The conversion only applied to call-phase reports, so a fixture that failed on every attempt still finished as a normal failure. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
Fixed in 53b61ec: the xfail conversion now applies to the terminal failed report in any phase, so a fixture that fails on every attempt ends as xfailed. Added a setup-failure regression test. |
|
Fixed — exhausted setup/teardown failures are now converted to xfail too, not just |
Summary
--xfail-flakyflag: a test that still fails after all its reruns are exhausted is reported asxfailedinstead offailed(functionality similar to pytest-ignore-flaky, as requested)execution_count > 1); tests failing on first attempt or excluded by--only-rerun/--rerun-exceptstill report as failedcallreport toskipped+wasxfail, so pytest's standard xfail reporting appliesCloses #92
Test plan
--reruns 2 --xfail-flaky→ 1 xfailed, 2 rerun, exit code 0--only-rerunnon-matching error still reports failed