Rust Iterator parity: dedup_with_count, scan, is_sorted, eq - #22
Open
virgesmith wants to merge 3 commits into
Open
Rust Iterator parity: dedup_with_count, scan, is_sorted, eq#22virgesmith wants to merge 3 commits into
virgesmith wants to merge 3 commits into
Conversation
Getting run lengths previously meant `chunk_by(lambda x: x).map(lambda x: (x[0], len(x[1])))`, which is clunky for what is a common operation, and materialises each run. `dedup_with_count` extends the existing `dedup` family (Rust's itertools crate pairs them the same way) and is the lazy, positional counterpart to `value_counts`: same `Itr[tuple[T, int]]` output shape, but counting adjacent runs rather than occurrences overall, preserving order, comparing by equality rather than requiring hashability, and staying lazy on infinite sources (provided no individual run is infinite). The (item, count) ordering deliberately matches `value_counts` rather than Rust's dedup_with_count, which yields (count, item); noted in a comment and the relnotes. Counts via `sum(1 for _ in g)` rather than `len(tuple(g))` so a run is consumed without being materialised. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #21, which GitHub closed when its head branch was renamed (
feat/dedup-with-count→feat/rust-iterator-parity) to match the widened scope. Same branch, same commits, plus two more.Four methods bringing
Itrcloser to Rust'sIteratortrait, each addressing a gap with no clean Python equivalent.dedup_with_count()— run-length encodingGetting the length of each consecutive run previously meant
chunk_by(lambda x: x).map(lambda x: (x[0], len(x[1])))— clunky, and it materialises every run just to measure it.dedupfamily, as Rust'sitertoolscrate pairs them, rather than inventing arun_lengths/rleconcept.(item, count)— deliberately the reverse of Rust'sdedup_with_count, to matchvalue_counts's existingItr[tuple[T, int]]shape. Consistency within the library beats consistency with Rust here; flagged in a comment, the docstring and the relnotes.value_counts: adjacent runs in source order vs occurrences overall. A test pins the divergence.sum(1 for _ in g)rather thanlen(tuple(g)), so a run is consumed without being materialised.scan(init, func)— stateful map with early exitThe one genuine capability gap against the trait:
accumulateyields intermediates but forces the state to share the items' type and cannot stop early.func(state, item)returns(new_state, output), orNoneto stop.Option; Python has no such type, but the halt signal here is the entire return value, not the output value — so(new_state, None)still yieldsNoneas an output and the two are never ambiguous. This is whyscanis safe to add wherefilter_mapwould not be (itsNone-as-drop sentinel is genuinely ambiguous whenNoneis a legitimate mapped value).is_sorted(key=None, *, reverse=False)No Python builtin does this;
list(it) == sorted(it)materialises twice and cannot short-circuit.keycovers Rust'sis_sorted_by_keyand matches themax/minsignature convention;reverseis an addition for symmetry withsorted_by, since checking descending order otherwise has no workaround short of materialising.eq(other)Element-wise comparison that short-circuits at the first difference, where
tuple(a) == tuple(b)materialises both sides. Documented explicitly:Itrdefines no__eq__, so==remains an identity check and.eq()is the content comparison. A test confirms it terminates against an infinite iterator.Not included, deliberately
filter_mapandfind_map(theNonesentinel is ambiguous in a language withoutOption—.map(f).filter(lambda x: x is not None)is explicit about that assumption at the call site), the comparison family beyondeq(cmp/lt/ge/partial_cmp— lexicographic ordering nobody reaches for in Python),fuse,advance_by,rposition,is_partitioned,intersperse_with, and everything untranslatable (by_ref,copied/cloned,collect_into,partition_in_place,size_hint, thetry_*family).Notes
is_sortedwidens the keyed sequence toIterable[Any]viacast, following the existingsorted_by/groupbyidiom:Tis unbounded, so it is not known to be orderable, andtycorrectly rejects<=on it.src/itrx/skill/SKILL.md(added in Add installable agent skill (itrx-skill) #20) is a third place a new public method must be recorded, alongsideREADME.mdanddoc/apidoc.md. All four methods are added to its lists, with gotcha entries for each.AGENTS.md's reviewer checklist item 8 still names only the README and apidoc — it should nameSKILL.mdtoo, or the shipped agent reference drifts silently, with nothing in the test suite to catch it. Left for a separate PR.Verification
ruff check,ruff format --check,ty check srcclean;pytest222 passed, coverage 100%.test_aggregation.pyandtest_transform_filter.py, covering empty/single-element/unhashable inputs, laziness on infinite sources, short-circuiting, and agreement with the neighbouring methods (dedup,sorted_by,value_counts).doc/apidoc.mdregenerated viauv run python src/scripts/introspect.py; README lazy/eager lists updated; relnotes entries under## Unreleased.🤖 Generated with Claude Code