Narwhals end tail imputer - #1013
Open
solegalli wants to merge 2 commits into
Open
Conversation
Shared base for the imputation module: _transform() (fit-state checks + column reorder) and transform() (fillna via imputer_dict_) are now dataframe-agnostic, with _get_feature_names_in() reading columns through narwhals on non-pandas input. Benchmarked the fillna step (select + fill from a per-column value dict) at 10k/100k/1M rows x 1/2/10 columns: pandas-native fillna runs ~1.3-1.6x faster than the narwhals-generic fill_null equivalent at the 10k-100k row sizes imputers are normally used at (the gap narrows to ~1.0x only past ~1M rows) - a real, not minimal, loss, so pandas keeps its own fast path (is_pandas = nwd.is_pandas_dataframe(X); if is_pandas is True: ... else narwhals fill_null per column). Also benchmarked a numpy rewrite (to_numpy + np.where per column, mirroring RelativeFeatures) but it did not beat pandas-native and was consistently slower than narwhals fill_null on polars, so it wasn't adopted here - unlike RelativeFeatures' arithmetic, a plain value fill is already close to a no-op for both pandas and narwhals/polars, leaving no room for a numpy win. The pandas<3 fillna-downcasting workaround (option_context + infer_objects) is preserved on the pandas branch but no longer imports pandas at module level - the module is fetched via nw.from_native(X).__native_namespace__() only once X is already confirmed to be a pandas dataframe, so no import is attempted on a polars-only install. Verified: tests/test_imputation full suite unchanged (95 passed, 7 pre-existing failures in test_check_estimator_imputers.py - sklearn's check_estimator feeds raw numpy arrays, which check_X() has always rejected per the narwhals migration's dataframe-only contract, predates this change). flake8 and mypy clean on the file. Module imports with pandas import blocked. sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
fit() now computes the Gaussian/IQR/max end-of-distribution values via a
single narwhals aggregation (nw_X.select(...) of per-variable mean/std/
quantile/max expressions), instead of pandas-only .mean()/.std()/.quantile().
transform() already worked cross-backend via the already-migrated
BaseImputer.
Merge vs split: benchmarked pandas-native vs narwhals-generic (on both
pandas and polars) at 10k/50k/100k rows x 1/2/10 columns, with NaNs present
(this is an imputer, so skip-NaN semantics matter - mean/std/quantile must
skip missing values like pandas' default skipna=True). Results:
- gaussian: narwhals-on-pandas is 0.93-1.5x pandas-native's time (parity
to a mild loss, narrowing towards 1.0x as rows scale up), and 3-10x
*faster* than pandas-native when run on polars.
- iqr: narwhals-on-pandas is consistently *faster* than pandas-native
(~1.3-2x), on both backends.
Nowhere near the "real loss" (1.7x+) split threshold, so one code path
(no is_pandas branching) serves both backends - unlike BaseImputer's
fillna, which stayed split because it *was* consistently 1.3-1.6x slower
via narwhals on pandas.
Also benchmarked a numpy rewrite (nanmean/nanstd/nanpercentile per column,
mirroring RelativeFeatures' numpy-acceleration pattern) and rejected it:
numpy's nan-aware reductions are slow (isnan-mask overhead), and at 10
columns narwhals-on-polars beat numpy-on-polars by ~10x (0.65ms vs 7.2ms
at 100k rows x 10 cols) since polars aggregates columns natively/in
parallel instead of looping in Python. RelativeFeatures' numpy win doesn't
transfer here because that transformer's arithmetic has no NaN-skipping
requirement, so plain (non-nan-aware) numpy ops sufficed there.
Tests rewritten to one parametrized test per behavior over
`@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame])`,
replacing the pandas-only test_end_tail_imputer.py. Test data uses `None`
for missing values instead of `np.nan`: polars treats a literal np.nan as
a real float (not a null), so it would NOT be skipped by mean/std/quantile
the way pandas skips NaN by default - `None` becomes a null on both
backends and is skipped consistently.
Docs: verified the existing house_prices example still runs and produces
matching output; added a "With polars" section to both the class
docstring and docs/user_guide/imputation/EndTailImputer.rst.
No bugs found in the pre-migration code. The 7 pre-existing
test_check_estimator_from_sklearn failures in this test module (numpy
array input now rejected by check_X, e.g. for MeanImputer) predate this
change and are unrelated to EndTailImputer.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.