Flaky: tracker flush and logging-output tests fail intermittently on master

[davidberenstein1957]

Three tests fail intermittently when the whole suite runs, across two files:

• tests/test_emissions_tracker_flush.py::test_carbon_tracker_offline_flush
• tests/test_logging_output.py::test_carbon_tracker_offline_logging_output

The failures are row counts (0 == 1, 2 == 3), so an expected measurement did not land before the assertion. Each test passes in isolation, and it reproduces on master, not just on feature branches.

The most likely cause is how CI runs the suite. .github/workflows/test-package.yml:36 is:

uv run codecarbon monitor -- uv run task test-coverage

so a live tracker wraps the entire test run. It holds /tmp/.codecarbon.lock and writes its own emissions.csv in the repo root for as long as the tests execute, while the tests under it are starting and stopping their own trackers. That is a real interaction and it is not present when you run a single test locally.

Two smaller things that make it worse:

• tests/test_logging_output.py builds three trackers with no output_dir and no output_file (lines 48-52, 64-69, and the @track_emissions decorator at 81-85), so they all default to emissions.csv in the current working directory. Nothing cleans that up.
• Both files use a fixed filename in tempfile.gettempdir() (emissions-test-TestCarbonTrackerFlush.csv, emissions-test-TestCarbonLoggingOutput.log) with no per-run suffix, and setUp/tearDown both delete it.

PeriodicScheduler chaining a threading.Timer per tick may also contribute, since a tracker a test does not stop keeps firing into the next test and tests/conftest.py only clears the hardware probe caches. Unverified. The re-arming timer goes away with #1339 / #1324, so it is worth re-measuring after one of those lands.

Likely fixes: give the logging test an explicit output_dir, make the temp filenames unique per process, and stop asserting on wall-clock-dependent row counts.

[davidberenstein1957]

Additional finding from a separate investigation, which narrows this considerably.

tests/test_emissions_tracker_flush.py and tests/test_logging_output.py are order/state-sensitive around a shared emissions.csv in the repository root. They fail on origin/master under some orderings and pass after rm emissions.csv.

So the first thing to try is not a scheduler fix but making these tests write to a per-test temporary directory instead of the repo root — and checking whether a stray emissions.csv is being left behind by an earlier test in the run.

That likely also explains why the failures are on row counts (0 == 1, 2 == 3): the assertions are counting rows in a file that a previous test already wrote to.

The leaked-timer theory above may still be a contributing factor, but this one is cheaper to check and easier to fix.

[davidberenstein1957]

Correcting my comment above — it got the mechanism wrong, and I have updated the issue body accordingly.

The claim was that tests/test_emissions_tracker_flush.py and tests/test_logging_output.py are sensitive to a shared emissions.csv in the repository root because they write their output there. They do not. On current master both already write into the system temp directory:

• tests/test_emissions_tracker_flush.py:26-30 — self.emissions_path = tempfile.gettempdir()
• tests/test_logging_output.py:27-31 — same

So "make these tests write to a temporary directory instead of the repo root" is not the fix; that part is already done.

What is actually shared:

1. Fixed filenames inside the temp directory. emissions-test-TestCarbonTrackerFlush.csv and emissions-test-TestCarbonLoggingOutput.log are constant paths, deleted in both setUp and tearDown. Two concurrent runs of the suite on the same machine (xdist, parallel CI jobs, a second local run) race for the same file, which produces exactly the missing-row assertions seen.
2. One tracker built with no output_dir. tests/test_logging_output.py:47-52 constructs EmissionsTracker(project_name=..., save_to_logger=True, logging_logger=...) without output_dir/output_file, so it writes emissions.csv into the current working directory — the repository root. That is where the stray root-level emissions.csv comes from, and nothing cleans it up. So the rm emissions.csv observation was real; the attribution was not.

Also worth flagging: tests/conftest.py only clears the hardware/TDP/GPU probe caches. It does not cancel or count leftover timers, so the leaked-scheduler theory in the original report remains untested rather than supported.