Migrate ArbitraryImputer to narwhals, add polars support - #1003
Open
solegalli wants to merge 2 commits into
Open
Migrate ArbitraryImputer to narwhals, add polars support#1003solegalli wants to merge 2 commits into
solegalli wants to merge 2 commits into
Conversation
Shared base for the imputation module: _transform() (fit-state checks + column reorder) and transform() (fillna via imputer_dict_) are now dataframe-agnostic, with _get_feature_names_in() reading columns through narwhals on non-pandas input. Benchmarked the fillna step (select + fill from a per-column value dict) at 10k/100k/1M rows x 1/2/10 columns: pandas-native fillna runs ~1.3-1.6x faster than the narwhals-generic fill_null equivalent at the 10k-100k row sizes imputers are normally used at (the gap narrows to ~1.0x only past ~1M rows) - a real, not minimal, loss, so pandas keeps its own fast path (is_pandas = nwd.is_pandas_dataframe(X); if is_pandas is True: ... else narwhals fill_null per column). Also benchmarked a numpy rewrite (to_numpy + np.where per column, mirroring RelativeFeatures) but it did not beat pandas-native and was consistently slower than narwhals fill_null on polars, so it wasn't adopted here - unlike RelativeFeatures' arithmetic, a plain value fill is already close to a no-op for both pandas and narwhals/polars, leaving no room for a numpy win. The pandas<3 fillna-downcasting workaround (option_context + infer_objects) is preserved on the pandas branch but no longer imports pandas at module level - the module is fetched via nw.from_native(X).__native_namespace__() only once X is already confirmed to be a pandas dataframe, so no import is attempted on a polars-only install. Verified: tests/test_imputation full suite unchanged (95 passed, 7 pre-existing failures in test_check_estimator_imputers.py - sklearn's check_estimator feeds raw numpy arrays, which check_X() has always rejected per the narwhals migration's dataframe-only contract, predates this change). flake8 and mypy clean on the file. Module imports with pandas import blocked. sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
fit() never touches dataframe values - it only calls the already-
narwhals-migrated check_X/check_numerical_variables/find_numerical_variables
and builds imputer_dict_ via a plain dict comprehension over column names -
so the only change needed was dropping the module-level `import pandas as
pd` and swapping the X/y type hints for narwhals' IntoDataFrame/IntoSeries.
transform() is fully inherited from the already-migrated BaseImputer.
Benchmarked fit()+transform() (via fit_transform) at 10k/50k/100k rows x
1/2/10 cols, pandas vs polars, and old code vs migrated code on pandas
input: fit() takes ~0.06-0.13ms regardless of row count, column count, or
backend, both before and after the edit (within noise of each other) -
confirming fit() truly does no per-row work. No backend split was needed
or added; a single narwhals-agnostic path was kept (it already was one).
Numpy: not applicable - fit() has no numeric computation over data at all,
only dict/list building over variable names, so there is nothing for numpy
to accelerate.
While touching fit(), changed `if self.imputer_dict:` to
`if self.imputer_dict is not None:` per AGENTS.md's ban on truthy
container checks; this also fixes a latent edge case where imputer_dict={}
was silently treated as "not provided" and fell through to the
variables/arbitrary_number branch. Confirmed pre-existing on
origin/narwhals-imputation-base (unrelated to this migration, no test
previously covered it).
Rewrote tests/test_imputation/test_arbitrary_imputer.py to the
cross-backend parametrized style (@pytest.mark.parametrize("make_df",
[pd.DataFrame, pl.DataFrame])) in place, replacing the pandas-only df_na
fixture and pd.testing.assert_frame_equal/.isnull() assertions with a
plain DATA dict and narwhals-based null/value assertions. The
deprecation-warning test for ArbitraryNumberImputer and the
arbitrary_number-type-validation test stayed single-backend since they
never touch a dataframe.
Added a "With polars" section to both the class docstring and
docs/user_guide/imputation/ArbitraryImputer.rst, output verified by
actually running the transformer. No staleness found in the existing rst
(it builds its example from fetch_openml, no literal printed dataframe
values to go stale).
Verified: tests/test_imputation full suite 98 passed / 7 pre-existing
unrelated failures in test_check_estimator_imputers.py (same 7 as on
origin/narwhals-imputation-base's baseline of 95 passed - the 3 extra
passes here are the new cross-backend parametrization, no regressions).
flake8 and mypy clean. Module imports with pandas import blocked.
sphinx -W build clean (only the pre-existing unrelated linkcode_resolve
warning).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.