Skip to content

Add dedup_with_count: lazy run-length encoding - #21

Closed
virgesmith wants to merge 3 commits into
mainfrom
feat/dedup-with-count
Closed

Add dedup_with_count: lazy run-length encoding#21
virgesmith wants to merge 3 commits into
mainfrom
feat/dedup-with-count

Conversation

@virgesmith

@virgesmith virgesmith commented Sep 1, 2026

Copy link
Copy Markdown
Owner

Why

Getting the length of each consecutive run currently means:

Itr([4, 4, 2, 3, 3, 1]).chunk_by(lambda x: x).map(lambda x: (x[0], len(x[1]))).collect()

That is clunky for a common operation — the x[0]/x[1] indexing exists only because chunk_by carries a key you don't need when grouping by identity — and it materialises every run into a tuple just to measure it.

Itr([4, 4, 2, 3, 3, 1]).dedup_with_count().collect()
# ((4, 2), (2, 1), (3, 2), (1, 1))

What

dedup_with_count() — lazily collapse each consecutive run of equal items into an (item, count) pair.

Design decisions

  • Extends the dedup family rather than introducing a run_lengths/rle concept — Rust's itertools crate pairs dedup with dedup_with_count the same way, and the two differ by exactly what the name says: dedup keeps the first of each run, dedup_with_count keeps it with the run's length.
  • (item, count), not Rust's (count, item) — consistency within this library beats consistency with Rust here: value_counts already returns Itr[tuple[T, int]], so the two now share an output shape and the only difference is which one you call. Flagged in a comment, the docstring and the relnotes, since it is a deliberate deviation.
  • Positions it as the lazy counterpart to value_countsvalue_counts counts occurrences over the whole iterator (eager, hashable items, most-common-first); dedup_with_count counts adjacent runs (lazy, equality-only, source order). On [4, 4, 2, 3, 3, 1] they give ((4,2),(2,1),(3,2),(1,1)) and ((4,2),(3,2),(2,1),(1,1)) respectively — covered by a test so the distinction stays pinned.
  • sum(1 for _ in g) rather than len(tuple(g)) — consumes each run without materialising it, which is the whole point over the chunk_by workaround.
  • Lazy, with one honest caveat — works on an infinite source, but an infinite individual run (e.g. itertools.repeat(1)) will hang, as it must. Stated in the docstring.

Verification

  • ruff check, ruff format --check, ty check src clean; pytest 202 passed, coverage 100%.
  • 8 new tests: basic, no duplicates, empty, single run, unhashable items, laziness on an infinite source, keys agreeing with dedup, and divergence from value_counts.
  • doc/apidoc.md regenerated via uv run python src/scripts/introspect.py; README lazy-methods list updated; relnotes entry under ## Unreleased.

Rebased on #20

Rebased onto main after #20 merged, resolving the expected ## Unreleased relnotes conflict by keeping both entries.

Note that #20 added a third place a new public method has to be recorded: the bundled src/itrx/skill/SKILL.md, which lists methods by category. This PR updates its lazy-method list, its Rust-parity list, and adds a dedup_with_count entry to the gotchas alongside dedup. AGENTS.md's reviewer checklist item 8 still says only "update README.md and regenerate doc/apidoc.md" — it should mention SKILL.md too, or the shipped agent reference will silently drift from the API. Not done here to keep this PR to one concern.

🤖 Generated with Claude Code

Getting run lengths previously meant
`chunk_by(lambda x: x).map(lambda x: (x[0], len(x[1])))`, which is clunky
for what is a common operation, and materialises each run.

`dedup_with_count` extends the existing `dedup` family (Rust's itertools
crate pairs them the same way) and is the lazy, positional counterpart to
`value_counts`: same `Itr[tuple[T, int]]` output shape, but counting
adjacent runs rather than occurrences overall, preserving order, comparing
by equality rather than requiring hashability, and staying lazy on infinite
sources (provided no individual run is infinite).

The (item, count) ordering deliberately matches `value_counts` rather than
Rust's dedup_with_count, which yields (count, item); noted in a comment and
the relnotes.

Counts via `sum(1 for _ in g)` rather than `len(tuple(g))` so a run is
consumed without being materialised.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@virgesmith
virgesmith force-pushed the feat/dedup-with-count branch from 283e68f to 0fa9e4e Compare September 1, 2026 08:27
@virgesmith

Copy link
Copy Markdown
Owner Author

Superseded by #22: the branch was renamed feat/dedup-with-countfeat/rust-iterator-parity to match its widened scope (now also scan, is_sorted and eq), and GitHub closed this PR on the rename rather than retargeting it. Same branch and commits, continued in #22.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant