migrate missing indicator to narwhals, add polar support - #1001
Open
solegalli wants to merge 2 commits into
Open
migrate missing indicator to narwhals, add polar support#1001solegalli wants to merge 2 commits into
solegalli wants to merge 2 commits into
Conversation
Shared base for the imputation module: _transform() (fit-state checks + column reorder) and transform() (fillna via imputer_dict_) are now dataframe-agnostic, with _get_feature_names_in() reading columns through narwhals on non-pandas input. Benchmarked the fillna step (select + fill from a per-column value dict) at 10k/100k/1M rows x 1/2/10 columns: pandas-native fillna runs ~1.3-1.6x faster than the narwhals-generic fill_null equivalent at the 10k-100k row sizes imputers are normally used at (the gap narrows to ~1.0x only past ~1M rows) - a real, not minimal, loss, so pandas keeps its own fast path (is_pandas = nwd.is_pandas_dataframe(X); if is_pandas is True: ... else narwhals fill_null per column). Also benchmarked a numpy rewrite (to_numpy + np.where per column, mirroring RelativeFeatures) but it did not beat pandas-native and was consistently slower than narwhals fill_null on polars, so it wasn't adopted here - unlike RelativeFeatures' arithmetic, a plain value fill is already close to a no-op for both pandas and narwhals/polars, leaving no room for a numpy win. The pandas<3 fillna-downcasting workaround (option_context + infer_objects) is preserved on the pandas branch but no longer imports pandas at module level - the module is fetched via nw.from_native(X).__native_namespace__() only once X is already confirmed to be a pandas dataframe, so no import is attempted on a polars-only install. Verified: tests/test_imputation full suite unchanged (95 passed, 7 pre-existing failures in test_check_estimator_imputers.py - sklearn's check_estimator feeds raw numpy arrays, which check_X() has always rejected per the narwhals migration's dataframe-only contract, predates this change). flake8 and mypy clean on the file. Module imports with pandas import blocked. sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…support
Removed the module-level `import pandas as pd`; X/y type hints now use
narwhals' IntoDataFrame/IntoSeries. This file overrides transform() rather
than extending BaseImputer's, so both the fit() null-count filter and the
transform() indicator-column step needed their own narwhals path.
Benchmarked both operations at 10k/50k/100k rows x 1/2/10 columns (varying
how many columns need indicators), plus a mixed string+numeric-dtype
dataset matching MissingIndicator's real "all variable types" usage:
- fit()'s `[var for var in variables_ if X[var].isnull().sum() > 0]` loop
is ~2-5x faster on pandas than a narwhals-generic `null_count()` call
(e.g. 100k rows x 10 cols: 0.41ms loop vs 0.78ms narwhals-on-pandas).
A vectorized `X[variables_].isnull().sum()` alternative didn't beat the
loop either. narwhals-on-polars was consistently fastest of all (its own
native path), so the split is pandas-loop vs narwhals-generic (used for
polars/other backends), matching BaseImputer's is_pandas branch pattern.
- transform()'s `X[vars].isna().astype("int8").add_suffix("_na")` +
`pd.concat` is ~2-5x faster on pandas than narwhals' with_columns
equivalent (100k rows x 10 cols: 0.28ms concat vs 1.27ms narwhals-on-
pandas), and also beats `assign()`-per-column (0.91ms) and `join()`
(0.44ms) alternatives - concat already batches all new columns in one
op. So transform() keeps the same pandas fast path, split from a
narwhals with_columns path for other backends.
Both losses are >1.7x, past the "keep pandas fast path" threshold, so
merging into one narwhals-generic path (as BaseImputer's docstring
discusses for its own fillna step) was not justified here either.
Numpy: converting columns via `.to_numpy()` + `pd.isna()` (the only numpy
op that works across MissingIndicator's mixed string/numeric columns,
since np.isnan raises on object arrays) was consistently ~1.7-2x slower
than pandas-native isnull()/isna() for both fit and transform on mixed
dtypes - the extra .to_numpy() copy plus pd.isna() dispatch outweighs any
gain, same conclusion as BaseImputer's fillna numpy experiment.
Tests: converted tests/test_imputation/test_missing_indicator.py from the
pandas-only `df_na` fixture to a plain DATA dict parametrized over
`make_df` in [pd.DataFrame, pl.DataFrame], asserting identical variables_
selection and identical `<var>_na` column values on both backends for the
same input (one cross-backend PerformanceWarning regression test stays
pandas-only, since it targets the pandas fast path specifically).
Docs: docs/user_guide/imputation/MissingIndicator.rst has no inline
printed output to go stale (it references a screenshot image instead of
doctest-style text) - verified its house_prices code example's logic
against the migrated transformer with a synthetic stand-in dataset (no
network access in this environment) and it behaves identically. Added a
verified "With polars" example to the class docstring.
Verified: tests/test_imputation/test_missing_indicator.py 29 passed.
tests/test_imputation full suite: 107 passed / 7 pre-existing failures
in test_check_estimator_imputers.py (confirmed identical failures against
a baseline run of origin/narwhals-imputation-base: 95 passed / same 7
failures - sklearn's check_estimator feeds raw numpy arrays, which
check_X() has always rejected per the narwhals migration's dataframe-only
contract; predates this change). flake8 and mypy clean. Module imports
with pandas import blocked. sphinx -W build clean (only the pre-existing
unrelated linkcode_resolve warning).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.