I know there is a number of approaches to preprocess training data with missing entries: dropping features, imputing mean values, etc.

I've compared few of such approaches and found that dropping features with missing values gives the lowest Mean Absolute Error for my training data. I applied it for the data and next trained a model with Random Forest Regressor.

Next, I wanted to evaluate the trained model for a test data that I haven't seen nor have access before. It came up that the test data contains different features with missing entries than the train data, i.e., features that have all entries in the train data, are missing some entries in the test data.

I wonder how can I handle such a situation?

  • Even if I apply same preprocessing to the test data, I will still have features with missing values. And Random Forest Regressor in scikit-learn complains when encountering missing entries.

  • If I drop all features from the test data that have missing entries and the trained model cannot be used because:

    Number of features of the model must match the input. Model n_features is 36 and input n_features is 25 

Is there a workaround for this preprocessing technique or it is a limitation of it?

  • 1
    $\begingroup$ It's obvious that you need to perform some sort of imputation in order to keep the set of features consistent. As you mentioned, replacing missing values with the column mean or median is one approach. I see no particular reason why removing a feature should yield a significantly better performance than any of these imputation techniques in context of random forests, unless the overwhelming majority of them are uncorrelated with the response. $\endgroup$ – bi_scholar May 24 at 11:09
  • $\begingroup$ @bi_scholar I guess because imputing mean values introduced more noise to data. For instance, there was GarageYrBlt feature with a year when a garage was built for a given real estate. It could be that null value has special meaning: no garage has been built at all for this estate (in fact, GarageCars GarageArea features have 0 where GarageYrBlt is null, confirming my assumption). Perhaps imputing some special value, e.g., 0 instead of years mean could give better results. Or replacing all garage columns with one saying whether a garage is present or not. $\endgroup$ – dzieciou May 24 at 12:39
  • 1
    $\begingroup$ While imputation most likely introduces noise (or rather some bias), this shouldn't hurt the random forest model, unless we are taking about a considerable fraction of missing values / imputed features. Random forests handle noisy or uninformative features quite well. Are you sure that the observed difference in performance is significant? Every dataset has its own oddities. $\endgroup$ – bi_scholar May 24 at 12:56
  • $\begingroup$ @bi_scholar You're right. Performance loss is not significant. MAE grows by \$225 (from \$17837 to \$18062) when switching from dropping features to imputing mean. We're talking here about house prices of around \$18,000 on average, $225 difference in prediction is really small. Regarding dataset: number of missing entries ranges from 5% to 17%, depending on the column. 23% of samples in the dataset has at least one column entry missing (seems like a lot for a dataset of 1460 samples). It might be that features that I dropped do not contribute much to the predicted house price. $\endgroup$ – dzieciou May 24 at 14:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.