Add dedup_with_count: lazy run-length encoding - #21
Closed
virgesmith wants to merge 3 commits into
Closed
Conversation
Getting run lengths previously meant `chunk_by(lambda x: x).map(lambda x: (x[0], len(x[1])))`, which is clunky for what is a common operation, and materialises each run. `dedup_with_count` extends the existing `dedup` family (Rust's itertools crate pairs them the same way) and is the lazy, positional counterpart to `value_counts`: same `Itr[tuple[T, int]]` output shape, but counting adjacent runs rather than occurrences overall, preserving order, comparing by equality rather than requiring hashability, and staying lazy on infinite sources (provided no individual run is infinite). The (item, count) ordering deliberately matches `value_counts` rather than Rust's dedup_with_count, which yields (count, item); noted in a comment and the relnotes. Counts via `sum(1 for _ in g)` rather than `len(tuple(g))` so a run is consumed without being materialised. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
virgesmith
force-pushed
the
feat/dedup-with-count
branch
from
September 1, 2026 08:27
283e68f to
0fa9e4e
Compare
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Getting the length of each consecutive run currently means:
That is clunky for a common operation — the
x[0]/x[1]indexing exists only becausechunk_bycarries a key you don't need when grouping by identity — and it materialises every run into a tuple just to measure it.What
dedup_with_count()— lazily collapse each consecutive run of equal items into an(item, count)pair.Design decisions
dedupfamily rather than introducing arun_lengths/rleconcept — Rust'sitertoolscrate pairsdedupwithdedup_with_countthe same way, and the two differ by exactly what the name says:dedupkeeps the first of each run,dedup_with_countkeeps it with the run's length.(item, count), not Rust's(count, item)— consistency within this library beats consistency with Rust here:value_countsalready returnsItr[tuple[T, int]], so the two now share an output shape and the only difference is which one you call. Flagged in a comment, the docstring and the relnotes, since it is a deliberate deviation.value_counts—value_countscounts occurrences over the whole iterator (eager, hashable items, most-common-first);dedup_with_countcounts adjacent runs (lazy, equality-only, source order). On[4, 4, 2, 3, 3, 1]they give((4,2),(2,1),(3,2),(1,1))and((4,2),(3,2),(2,1),(1,1))respectively — covered by a test so the distinction stays pinned.sum(1 for _ in g)rather thanlen(tuple(g))— consumes each run without materialising it, which is the whole point over thechunk_byworkaround.itertools.repeat(1)) will hang, as it must. Stated in the docstring.Verification
ruff check,ruff format --check,ty check srcclean;pytest202 passed, coverage 100%.dedup, and divergence fromvalue_counts.doc/apidoc.mdregenerated viauv run python src/scripts/introspect.py; README lazy-methods list updated; relnotes entry under## Unreleased.Rebased on #20
Rebased onto
mainafter #20 merged, resolving the expected## Unreleasedrelnotes conflict by keeping both entries.Note that #20 added a third place a new public method has to be recorded: the bundled
src/itrx/skill/SKILL.md, which lists methods by category. This PR updates its lazy-method list, its Rust-parity list, and adds adedup_with_countentry to the gotchas alongsidededup.AGENTS.md's reviewer checklist item 8 still says only "update README.md and regenerate doc/apidoc.md" — it should mentionSKILL.mdtoo, or the shipped agent reference will silently drift from the API. Not done here to keep this PR to one concern.🤖 Generated with Claude Code