Narwhals mean median imputer - #1015
Open
solegalli wants to merge 2 commits into
Open
Conversation
Shared base for the imputation module: _transform() (fit-state checks + column reorder) and transform() (fillna via imputer_dict_) are now dataframe-agnostic, with _get_feature_names_in() reading columns through narwhals on non-pandas input. Benchmarked the fillna step (select + fill from a per-column value dict) at 10k/100k/1M rows x 1/2/10 columns: pandas-native fillna runs ~1.3-1.6x faster than the narwhals-generic fill_null equivalent at the 10k-100k row sizes imputers are normally used at (the gap narrows to ~1.0x only past ~1M rows) - a real, not minimal, loss, so pandas keeps its own fast path (is_pandas = nwd.is_pandas_dataframe(X); if is_pandas is True: ... else narwhals fill_null per column). Also benchmarked a numpy rewrite (to_numpy + np.where per column, mirroring RelativeFeatures) but it did not beat pandas-native and was consistently slower than narwhals fill_null on polars, so it wasn't adopted here - unlike RelativeFeatures' arithmetic, a plain value fill is already close to a no-op for both pandas and narwhals/polars, leaving no room for a numpy win. The pandas<3 fillna-downcasting workaround (option_context + infer_objects) is preserved on the pandas branch but no longer imports pandas at module level - the module is fetched via nw.from_native(X).__native_namespace__() only once X is already confirmed to be a pandas dataframe, so no import is attempted on a polars-only install. Verified: tests/test_imputation full suite unchanged (95 passed, 7 pre-existing failures in test_check_estimator_imputers.py - sklearn's check_estimator feeds raw numpy arrays, which check_X() has always rejected per the narwhals migration's dataframe-only contract, predates this change). flake8 and mypy clean on the file. Module imports with pandas import blocked. sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fit's mean()/median() computation is split by backend and, on the pandas branch, additionally rewritten to use NumPy directly. Benchmarked (10k-100k rows x 1-10 cols): narwhals-on-pandas vs pandas-native .mean()/.median() showed the same real, not minimal, loss (1.0-3.0x) already documented for BaseImputer's fillna and CategoricalImputer's mode(), so pandas keeps its own fast path. Going further, benchmarked a bulk NumPy nanmean/nanmedian pass (to_numpy() + axis=0 reduction, mirroring MathFeatures' reducer pattern) against pandas-native .mean()/.median() and found NumPy consistently as fast or faster (ratios 0.5-1.05x) - a real win, so the pandas branch now uses NumPy instead of pandas' own methods. For polars, the equivalent NumPy round-trip was benchmarked too and lost to narwhals' native per-column mean()/median() expressions (1.8-3.5x slower for mean; mixed but trending slower for median at scale), so the polars/narwhals branch computes stats with a single narwhals select() of one expression per variable instead - benchmarked against a per-column loop and against select()+to_native().to_dicts() and found select()+rows(named=True) is equal-or-faster and backend-agnostic (no reliance on a polars-only to_dicts() method). All-NaN/all-null columns produce matching values on both backends (verified directly): NumPy's nanmean/nanmedian warn on all-NaN slices where pandas' methods don't, so those warnings are suppressed the same way MathFeatures does. Nullable extension dtypes that would produce object arrays fall back to pandas' native .mean()/.median(), same guard as MathFeatures' dtype.kind check. Found and fixed a real crash: narwhals' select() with zero expressions collapses row count to 0 too, so stats.rows(named=True)[0] would IndexError when return_empty=True yields no numerical variables on polars input. Added an explicit empty-variables guard that skips the backend branch entirely instead of relying on backend-specific zero-column behaviour. Rewrote tests as one parametrized test per behaviour over pd.DataFrame/pl.DataFrame (a self-contained DATA dict replacing the pandas-only df_na fixture, matching the CategoricalImputer migration's pattern), keeping the MeanImputer/MeanMedianImputer deprecation-warning parametrization on top. Verified: tests/test_imputation full suite - 99 passed (up from 95 pre-migration, same tests plus new polars parametrizations), same 7 pre-existing failures in test_check_estimator_imputers.py (sklearn's check_estimator feeds raw numpy arrays, rejected by check_X's dataframe-only contract from the base migration - confirmed identical root cause against the pre-migration baseline via git stash). flake8 and mypy clean. mean_median.py's actual import chain (base_imputer, dataframe_checks, variable_handling) verified pandas-free with pandas blocked, using direct module loading to bypass the sibling not-yet-migrated imputers in imputation/__init__.py. sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning). Every doc example (docstring pandas/polars examples and the new "With polars" section in MeanImputer.rst) re-run against live output. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.