Skip to content

Fix coverage double-counting (src/ vs installed package)#22

Merged
Lasse Borgholt (borgholt) merged 5 commits into
mainfrom
fix-coverage-double-counting
Jun 23, 2026
Merged

Fix coverage double-counting (src/ vs installed package)#22
Lasse Borgholt (borgholt) merged 5 commits into
mainfrom
fix-coverage-double-counting

Conversation

@borgholt

Copy link
Copy Markdown
Collaborator

Problem

After #21, reported coverage dropped from ~93% to ~50%. This is a measurement artifact, not a real loss of test coverage.

The CI coverage table lists every module twice: the executed copy under …/site-packages/error_align/* (real coverage) and an untouched copy under src/error_align/* reported at 0%. The phantom src/ copy has the same statement count as the real package with all statements missed, so it doubles the denominator while hits stay flat — mechanically halving the percentage (645 hit / 645 → 634/1290 ≈ 49%).

Root cause

Two coverage sources were tracked at once:

  • --cov=src (from pytest.ini)
  • --cov=error_align (from the test workflow)

poetry run pip install . installs the package non-editably (this is required — it compiles the C++ extension, so it is not redundant). Imports then resolve to site-packages, while --cov=src keeps measuring the on-disk src/ tree at 0%. While an editable install dominated, both resolved to the same paths and coverage deduped them; the 0.1.0b9 → 0.1.0b10 version bump made pip install . actually lay down the non-editable copy (previously a no-op), exposing the double count.

Fix

  • pytest.ini: measure a single source — --cov=error_align instead of --cov=src.
  • pyproject.toml: add [tool.coverage.paths] mapping src/error_align*/site-packages/error_align (plus [tool.coverage.run] source) so the two locations merge into one logical path regardless of editable vs non-editable install.

The C++-compiling pip install . step is left intact.

Verification

Local run now shows a single copy per module, TOTAL 94% (non-baseline), 19 passed — no src/ + site-packages duplication.

🤖 Generated with Claude Code

CI installs the package non-editably (to compile the C++ extension), so
imports resolve to site-packages while --cov=src separately measured the
untouched src/ tree at 0% — doubling the denominator and roughly halving
reported coverage (~93% -> ~50%).

Measure a single source (--cov=error_align) and add [tool.coverage.paths]
to merge the src/ and site-packages copies into one logical path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented Jun 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.31%. Comparing base (9c5a1d4) to head (59d5334).

Additional details and impacted files
@@             Coverage Diff             @@
##             main      #22       +/-   ##
===========================================
+ Coverage   50.47%   93.31%   +42.83%     
===========================================
  Files          14        9        -5     
  Lines        1369      658      -711     
  Branches      229      106      -123     
===========================================
- Hits          691      614       -77     
+ Misses        651       17      -634     
  Partials       27       27               

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the project’s pytest/coverage configuration to prevent coverage double-counting when tests execute the installed error_align package (in site-packages) while also tracking the src/ tree.

Changes:

  • Switch pytest coverage collection from --cov=src to --cov=error_align to avoid measuring an unexecuted src/ copy at 0%.
  • Add Coverage.py path-mapping in pyproject.toml to unify src/error_align and */site-packages/error_align into a single logical source location.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
pytest.ini Updates default pytest addopts to measure only error_align (not src).
pyproject.toml Adds Coverage.py path mapping (and run source) intended to merge editable/non-editable install paths.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread pyproject.toml Outdated
The baselines/ modules (POWER, etc.) are optional and largely untested.
They only started counting toward coverage when rapidfuzz became a core
dependency (#21), since that made error_align.baselines importable in CI —
dropping reported coverage from ~93% to ~50%. The codecov.yml ignore glob
("src/error_align/baselines/*") never matched their installed path
("error_align/baselines/..."), so they slipped through.

Omit them at the coverage layer (source-independent) and fix the codecov
ignore glob to match any baselines path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
.coveragerc takes precedence over pyproject.toml, so the [tool.coverage.*]
config added earlier was silently ignored — two configs that could diverge.
Remove it and keep .coveragerc as the single source of truth.

Also fix its omit glob: src/error_align/baselines/* only matched the editable
layout, so in CI (site-packages) baselines were not omitted by coverage.py and
were excluded only by the codecov ignore. Use */baselines/* to match both.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@borgholt Lasse Borgholt (borgholt) merged commit 0131901 into main Jun 23, 2026
9 checks passed
@borgholt Lasse Borgholt (borgholt) deleted the fix-coverage-double-counting branch June 23, 2026 09:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants