How to return feature_names_out with sklearn.preprocessing.FunctionTransformer? [closed]

Question

My goal is to impute not with sklearn.impute.SimpleImputer. My goal is to impute with sklearn.preprocessing.FunctionTransformer. This is because in some cases, it is obvious that we should fill a column based on other column and not with median or mean.

The problem is, I can't call get_feature_names_out

What I've tried:

1. I tried to use the parameter feature_names_out. However, the input_features is set to None.
2. I thought input_features is set to None because I did not set validate=True. But, when it throws error.

ValueError: Input X contains NaN.
FunctionTransformer does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values

The code

import pandas as pd
import numpy as np
from sklearn.preprocessing import FunctionTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline

df = pd.DataFrame(
    data={
        'PAYMENTS': [np.nan, 100, np.nan],
        'MINIMUM_PAYMENTS': [100, 100, 100]
    }
)
df

def impute(df):
    df['PAYMENTS'].fillna(df['MINIMUM_PAYMENTS'], inplace=True)
    return df

def f(self, input_features):
    print(input_features)
    return input_features

pipe = Pipeline(
    steps=[
        ('imputer', FunctionTransformer(func=impute, feature_names_out=f))
        # ('imputer', SimpleImputer(strategy='median'))
    ]
)

pipe.fit_transform(X=df), 
pipe.get_feature_names_out()

Answer 1

I thought input_features is set to None because I did not set validate=True.

That is indeed the case. The docs imply it, based on the descriptions of other attributes (e.g feature_names_in_), but don't state it explicitly. Adding a note about this behavior might be an easy and useful contribution to Scikit-learn.

As for the error message... unfortunately that's a baked-in part of the validate=True logic. This validation logic is generic and re-used in several different places. Its implementation is in sklearn.utils.validation._assert_all_finite.

This seems to be an oversight/bug in FeatureTransformer. There is an allow_nan= option in this internal function, but the user is not given an opportunity to change its value. The validation logic in FeatureTransformer eventually flows to sklearn.utils.validation.check_array, which calls _assert_all_finite with allow_nan=False unconditionally.

As with the omission in the docs, I think the only solution here (other than your workaround) is to file an issue on the Scikit-learn issue tracker.

It's very likely that they never considered using FunctionTransformer for imputation like this, since imputation is very often not "stateless", and FunctionTransformer is meant to be used for "stateless" transformations.