Narwhals categorical imputer - #1009
Open
solegalli wants to merge 2 commits into
Open
Conversation
Shared base for the imputation module: _transform() (fit-state checks + column reorder) and transform() (fillna via imputer_dict_) are now dataframe-agnostic, with _get_feature_names_in() reading columns through narwhals on non-pandas input. Benchmarked the fillna step (select + fill from a per-column value dict) at 10k/100k/1M rows x 1/2/10 columns: pandas-native fillna runs ~1.3-1.6x faster than the narwhals-generic fill_null equivalent at the 10k-100k row sizes imputers are normally used at (the gap narrows to ~1.0x only past ~1M rows) - a real, not minimal, loss, so pandas keeps its own fast path (is_pandas = nwd.is_pandas_dataframe(X); if is_pandas is True: ... else narwhals fill_null per column). Also benchmarked a numpy rewrite (to_numpy + np.where per column, mirroring RelativeFeatures) but it did not beat pandas-native and was consistently slower than narwhals fill_null on polars, so it wasn't adopted here - unlike RelativeFeatures' arithmetic, a plain value fill is already close to a no-op for both pandas and narwhals/polars, leaving no room for a numpy win. The pandas<3 fillna-downcasting workaround (option_context + infer_objects) is preserved on the pandas branch but no longer imports pandas at module level - the module is fetched via nw.from_native(X).__native_namespace__() only once X is already confirmed to be a pandas dataframe, so no import is attempted on a polars-only install. Verified: tests/test_imputation full suite unchanged (95 passed, 7 pre-existing failures in test_check_estimator_imputers.py - sklearn's check_estimator feeds raw numpy arrays, which check_X() has always rejected per the narwhals migration's dataframe-only contract, predates this change). flake8 and mypy clean on the file. Module imports with pandas import blocked. sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fit's mode() computation is split by backend: benchmarked (10k-100k rows x 1-10 cols) narwhals-on-pandas against pandas-native mode() and found a real, not minimal, 1.4-1.7x loss, consistent with BaseImputer's earlier split decision for fillna - so pandas keeps calling its own .mode(). Also benchmarked pandas' per-column mode() loop against its original batch X[variables_].mode() call and found no advantage to the batch form (ratios 0.77-0.97x), so both backends now share one per-variable loop structure, just with a different mode() call inside - simpler than the original single-var/multi-var split without losing performance. Found and fixed a real mode-tie bug: polars' native mode() does not drop nulls first (pandas' does, by default), so a column whose nulls outnumber any single category would make null "the mode" on polars instead of raising the multi-mode ValueError pandas raises. Fixed by calling drop_nulls() before mode(keep="all") on the narwhals branch; verified both backends now raise on the same tied columns and agree on the same single mode when there's no tie. Investigated pandas' category dtype vs polars' Categorical/Enum, since they aren't equivalent APIs. polars' Categorical auto-widens on fill_null (no add_categories-equivalent step needed, unlike pandas' category dtype which still needs the existing add_categories call or it raises TypeError). polars' Enum has a genuinely fixed category set: filling it with a value outside that set silently writes null instead of erroring - confirmed this is real, not hypothetical, so added an explicit check that raises a clear ValueError instead of corrupting data silently. Also confirmed polars never silently upcasts a string-typed column back to numeric the way pandas' fillna+ infer_objects does, so return_object is a documented no-op there. Rewrote tests as one parametrized test per behavior over pd.DataFrame/pl.DataFrame, using a shared DATA dict instead of the pandas-only df_na fixture. Kept pandas' object-dtype-for-numeric-vars tests and the category-dtype tests single-backend (genuinely pandas-specific dtype quirks with no polars equivalent), and added new single-backend polars tests for Categorical widening and the Enum fixed-category error path. Verified: tests/test_imputation full suite unchanged except for the new cases (105 passed, same 7 pre-existing failures in test_check_estimator_imputers.py that predate this change, per BaseImputer's migration). flake8 and mypy clean. Module's own import chain (dataframe_checks, variable_handling, base_imputer) verified pandas-free with pandas blocked - the whole feature_engine.imputation package still imports pandas only because sibling imputers are not yet migrated. Every doc example re-run against the live house_prices dataset and a pandas dtype-name string fixed to match pandas 3's actual output; added a "With polars" section with the Enum caveat. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.