My goal is to impute not with sklearn.impute.SimpleImputer. My goal is to impute with sklearn.preprocessing.FunctionTransformer. This is because in some cases, it is obvious that we should fill a column based on other column and not with median or mean.

The problem is, I can't call get_feature_names_out

What I've tried:

  1. I tried to use the parameter feature_names_out. However, the input_features is set to None.
  2. I thought input_features is set to None because I did not set validate=True. But, when it throws error.
ValueError: Input X contains NaN.
FunctionTransformer does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values

The code

import pandas as pd
import numpy as np
from sklearn.preprocessing import FunctionTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline

df = pd.DataFrame(
        'PAYMENTS': [np.nan, 100, np.nan],
        'MINIMUM_PAYMENTS': [100, 100, 100]

def impute(df):
    df['PAYMENTS'].fillna(df['MINIMUM_PAYMENTS'], inplace=True)
    return df

def f(self, input_features):
    return input_features

pipe = Pipeline(
        ('imputer', FunctionTransformer(func=impute, feature_names_out=f))
        # ('imputer', SimpleImputer(strategy='median'))

  • $\begingroup$ The current solution that I found is set the parameter validate=False, define a function def f(self): return self.kw_args['features']. However, I believe this is a hack and there is correct way to do it. I hope someone can point it out. $\endgroup$ Nov 26, 2022 at 4:13
  • $\begingroup$ Note that this might be more of a "programming question" than a "stats question". It might be more likely to get an answer on Stackoverflow. $\endgroup$ Nov 26, 2022 at 5:42
  • $\begingroup$ Also, you almost certainly do not want to use inplace=True. That could modify the original data frame that was passed into the pipeline! $\endgroup$ Nov 26, 2022 at 5:43
  • $\begingroup$ @shadowtalker Noted, from now on I will write code problem on StackOverflow. I thought Cross-Validated was for everything related to Data Science. Anyway, thank you for the inplace=True. I checked it and it does change the original DataFrame. However, I found a weird behavior, if I use ColumnTransformer on top of 2 pipe, the original DataFrame will not be modified. $\endgroup$ Nov 26, 2022 at 6:14
  • $\begingroup$ It's possible that ColumnTransformer has some internal behavior that results in copying the data frame. This site is great for statistics questions, but questions purely about code and programming have usually been redirected to StackOverflow. $\endgroup$ Nov 26, 2022 at 6:19

1 Answer 1


I thought input_features is set to None because I did not set validate=True.

That is indeed the case. The docs imply it, based on the descriptions of other attributes (e.g feature_names_in_), but don't state it explicitly. Adding a note about this behavior might be an easy and useful contribution to Scikit-learn.

As for the error message... unfortunately that's a baked-in part of the validate=True logic. This validation logic is generic and re-used in several different places. Its implementation is in sklearn.utils.validation._assert_all_finite.

This seems to be an oversight/bug in FeatureTransformer. There is an allow_nan= option in this internal function, but the user is not given an opportunity to change its value. The validation logic in FeatureTransformer eventually flows to sklearn.utils.validation.check_array, which calls _assert_all_finite with allow_nan=False unconditionally.

As with the omission in the docs, I think the only solution here (other than your workaround) is to file an issue on the Scikit-learn issue tracker.

It's very likely that they never considered using FunctionTransformer for imputation like this, since imputation is very often not "stateless", and FunctionTransformer is meant to be used for "stateless" transformations.

  • $\begingroup$ I am 8 weeks in learning about Data Analysis, and Machine Learning. since imputation is very often not "stateless" do you mean that ML engineer do not handle missing values of a column using the values from other column (ie. they usually use stateful like the mean value of that column from the train set)? $\endgroup$ Nov 26, 2022 at 6:11
  • $\begingroup$ Or do you mean, ML Engineer can handle missing values of a column using the value from other column. It just that ML Engineer do not use 'FunctionTransformer'. If so, can you suggest me the usual way? $\endgroup$ Nov 26, 2022 at 6:12
  • $\begingroup$ @kidfrom I would not try to generalize to what ML Engineers do in general. But it appears that the Scikit-learn developers specifically did not consider it when they wrote this code. It's possible that this has already been discussed on their mailing list or issue tracker. I suggest searching and opening an issue if you don't see an existing discussion. $\endgroup$ Nov 26, 2022 at 6:13

Not the answer you're looking for? Browse other questions tagged or ask your own question.