Migrate GeoDistanceFeatures to narwhals, add polars support - #993
Merged
Conversation
Six pandas-specific spots split into a pandas-native branch and a narwhals-generic branch, each decision benchmarked at 10k-50k rows and 0/1/6 extra columns (not assumed): - missing-columns check, feature_names_in_ extraction: narwhals-on-pandas is 13-22x slower (pure metadata overhead, row-count independent) - kept the pandas fast path established in Pass 1. - coordinate range validation: 6-8.6x slower on narwhals-on-pandas - new narwhals branch added (previously crashed outright on polars), pandas branch untouched. - numpy extraction of the 4 coordinate columns: 5-9x slower via narwhals on pandas; for the narwhals branch itself, .get_column().to_numpy() per column beats .select().to_numpy() by 5-7x on polars, so that's what it uses. - assign new column + optional drop: 1.7-2.9x slower on narwhals-on-pandas, consistent with the bar CyclicalFeatures used to keep branches separate. - column reorder is the one exception - narwhals-on-pandas is actually ~35% *faster* here at 10k rows - but stays a two-branch split per an explicit decision to keep the narwhals-everywhere pattern consistent with Pass 1/2, rather than special-case one operation. Verified end-to-end (not just isolated snippets): pandas output identical to the pre-migration code, polars value-identical to pandas, ~2% pandas speed delta (noise) at 10k rows/1 extra column, both backends' fit() error paths (missing columns, out-of-range coordinates) raise the same messages. Also fixed a pre-existing, unrelated inaccuracy in the class docstring's Examples section - the documented pandas output didn't match what the current (pre-migration) code actually produces. The same drift exists in the user guide's Python-implementation number tables (haversine, euclidean, manhattan, miles) but fixing those throughout is out of scope for this pass - flagged separately. Tests parametrized pandas+polars where a dataframe is involved; pure __init__/tag-validation tests (no dataframe) left as-is, already using match= throughout.
solegalli
commented
Aug 24, 2026
Every numeric output table in the "Python implementation" section (haversine, euclidean, manhattan, miles) had drifted from what the code actually produces - confirmed by running each documented example directly and comparing. Some differences are rounding-level, but euclidean trip 4 (1720.18 documented vs 1898.82 actual) and manhattan trip 2 (4684.16 vs 4266.82) are real gaps, and the pipeline predictions example was the furthest off: documented as the training targets exactly ([100, 150, 80, 200]), actual output is [116.67, 120.75, 88.48, 204.10]. Pre-existing, unrelated to the narwhals migration - verified the old, unmigrated code produces the same "actual" numbers used here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Six pandas-specific spots split into a pandas-native branch and a narwhals-generic branch, each decision benchmarked at 10k-50k rows and 0/1/6 extra columns (not assumed):
Verified end-to-end (not just isolated snippets): pandas output identical to the pre-migration code, polars value-identical to pandas, ~2% pandas speed delta (noise) at 10k rows/1 extra column, both backends' fit() error paths (missing columns, out-of-range coordinates) raise the same messages.
Also fixed a pre-existing, unrelated inaccuracy in the class docstring's Examples section - the documented pandas output didn't match what the current (pre-migration) code actually produces. The same drift exists in the user guide's Python-implementation number tables (haversine, euclidean, manhattan, miles) but fixing those throughout is out of scope for this pass - flagged separately.
Tests parametrized pandas+polars where a dataframe is involved; pure init/tag-validation tests (no dataframe) left as-is, already using match= throughout.