Skip to content

Add pytest-reportlog integration test and document compatibility #379

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
LouisDeconinck wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Add pytest-reportlog integration test and document compatibility #379

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e3f4f19
Add pytest-reportlog integration test and document compatibility
LouisDeconinck Sep 17, 2026
9f38671
Merge branch 'master' into issue-134-reportlog
icemac Sep 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -366,6 +366,10 @@ be marked as failed.
Compatibility
-------------

* This plugin is compatible with
`pytest-reportlog <https://pypi.org/project/pytest-reportlog/>`_: each
attempt is emitted as a ``TestReport``, with retried failures using the
``"rerun"`` outcome.

@icemac icemac Sep 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

— Comment created by Claude

The user-facing part of this PR is this new compatibility note. The only change log fragment is 134.misc.rst, and it describes the test setup. If the note should show up under Documentation in the change log, consider adding a 134.doc.rst fragment, e.g. Document compatibility with pytest-reportlog.

* This plugin is *not* compatible with pytest-xdist's --looponfail flag.
* This plugin is *not* compatible with
`pytest-forked <https://pypi.org/project/pytest-forked/>`_: both plugins
Expand Down
3 changes: 3 additions & 0 deletions changes/134.misc.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Add an integration test covering ``pytest-reportlog`` compatibility: each

@icemac icemac Sep 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

— Comment created by Claude

changes/README.rst says: "Do not hard wrap the text of an entry: towncrier re-wraps each paragraph to fit the change log, and hard wrapping produces a ragged result." This entry is hard wrapped across three lines. It also repeats the README wording. A single short line is enough:

Suggested change
Add an integration test covering ``pytest-reportlog`` compatibility: each
Add an integration test for ``pytest-reportlog`` compatibility.

attempt is emitted as a ``TestReport``, with retried failures using the
``"rerun"`` outcome.
23 changes: 23 additions & 0 deletions tests/test_pytest_rerunfailures.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -617,6 +617,29 @@ def test_pass():
assert_outcomes(result, passed=1, rerun=2)


def test_reportlog_contains_rerun_reports(testdir):
pytest.importorskip("pytest_reportlog")
testdir.makepyfile(
f"""
def test_pass():
{temporary_failure()}"""
)
result = testdir.runpytest("--reruns", "1", "--report-log=log.jsonl")
assert_outcomes(result, passed=1, rerun=1)

import json

@icemac icemac Sep 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

— Comment created by Claude

Minor: import json is function-local. The rest of the module imports stdlib modules at the top (import random, import re, ...). Please move it up there.


reports = [
json.loads(line)
for line in testdir.tmpdir.join("log.jsonl").read().splitlines()
if line.strip()
]
test_reports = [r for r in reports if r.get("$report_type") == "TestReport"]
call_reports = [r for r in test_reports if r.get("when") == "call"]

@icemac icemac Sep 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

— Comment created by Claude

The test only checks when == "call" reports. The README now says each attempt is emitted as a TestReport, but these cases are not covered:

  • setup/teardown reports of the rerun attempt
  • xdist crash reruns, where the rerun report from pytest_handlecrashitem has when == "???". I checked this manually and it currently works.

A regression in those paths would not be caught. Either check all TestReports for the item, or narrow the README wording.

assert [r["outcome"] for r in call_reports] == ["rerun", "passed"]
assert [r["rerun"] for r in call_reports] == [0, 1]


def test_flaky_marker_with_zero_reruns_disables_rerun(testdir):
testdir.makepyfile(
f"""
Expand Down
1 change: 1 addition & 0 deletions tox.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ minversion = 4.0
commands = pytest tests/ {posargs}
deps =
xdist: pytest-xdist
pytest-reportlog

@icemac icemac Sep 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

— Comment created by Claude

This only affects local tox runs. The GitHub Actions jobs in .github/workflows/test.yml don't use tox. They run pip install pytest / pip install -e ., so pytest-reportlog is never installed there and test_reportlog_contains_rerun_reports is always skipped by importorskip. The behavior this PR wants to pin is therefore never checked in CI. Please add pytest-reportlog to the pip install steps in test.yml, or at least to the main matrix job.

@icemac icemac Sep 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes pytest-reportlog a dependency for all tests, which I do not like.
I'd prefer a conditional dependency like it is done for xdist, but we should combine xdist and reflog into one group "deps" so we do not have a runner for each dependency. Also update the GitHub workflow accordingly.

— 100 % human review, no AI involved: Weight higher than the AI generated comments.

pytest82: pytest==8.2.*
pytest83: pytest==8.3.*
pytest84: pytest==8.4.*
Expand Down
Loading