I'm trying to debug something, and I think Imputer might be causing me some problems.

When using Imputer in a cross-validation setting, what happens if the test set has no missing values, but (in an extreme case), the validation set has some column that is completely missing. I believe Imputer will then remove that entire column. Then the classifier will be given an input of a lesser dimension?

I think the reverse case can also lead to problems (training data has a completely missing column, but validation data is complete).

How does Imputer deal with these situations?

My usage is something like this:

imputer = Imputer(missing_values="NaN", axis=0,  strategy="mean", verbose=5)
pipe = Pipeline(steps=[("imputer", imputer), ('my_classifier', my_classifier)])
gs = GridSearchCV(pipe, cv=5)
gs.fit(x_train, y_train)

closed as off-topic by dsaxton, John, Sycorax, Christoph Hanck, gung Mar 11 '16 at 8:08

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – dsaxton, John, Sycorax, Christoph Hanck, gung
If this question can be reworded to fit the rules in the help center, please edit the question.


In the first case

"test set has no missing values, but (in an extreme case), the validation set has some column that is completely missing"

It should not create any problem since the strategy for the imputer is replace by mean, meaning, each missing value get replaced by the mean of the column. If the imputer is fit on the first set (training set) than the mean of the column is known and the imputer can replace each NaN with this value. The interesting part is when the imputer is fitted on a case where a complete column is missing, in that case the mean of the column would be NaN and for sure would cause a problem, it would indeed remove your column and cause an error later in your pipeline.

  • $\begingroup$ The docs say "When axis=0, columns which only contained missing values at fit are discarded upon transform." $\endgroup$ – Fequish Sep 20 '15 at 18:34
  • $\begingroup$ aaah thank you, my mistake! That is not so nice indeed. $\endgroup$ – Bas van Stein Sep 20 '15 at 18:35
  • $\begingroup$ Actually, doesn't this mean that if a test column is all NaN, then essentially that column will be removed from both the test and validation sets? I'm not sure how to interpret it... $\endgroup$ – Fequish Sep 20 '15 at 19:06
  • $\begingroup$ Could be, I have to run some tests to verify this, interesting question at least :) thanks $\endgroup$ – Bas van Stein Sep 20 '15 at 19:07

Not the answer you're looking for? Browse other questions tagged or ask your own question.