1
$\begingroup$

I always struggled to get imputation for missing values right, and it doesn't help that you can find contradictory opinions online about it.

Say I have data X that I split into X_train and X_test, and I want to use a machine learning algorithm that does not handle missing values (say logistic regression or a neural network).

My approach right now is to fit the standard_scaler of sklearn to my train data X_train, then use that fitted scaler to transform my test_data X_test, then fill missing values of both with 0s. The standard scaler subtracts the mean of the training samples and divides by the samples standard deviation.

I combine the resulting X_train, X_tet via concatenation to get a new X (to use when evaluating etc.).

In the past, I simply used the standard scaler on X before doing a train_test split, filled nas with zeros and went along with my machine learning. It seemed to me that was wrong, as I used information of my test data to fit the standard scaler. Is the approach described above reasonable and does it avoid potential pitfalls?

$\endgroup$

1 Answer 1

1
$\begingroup$

Firstly, the approach you describe (apply a scaling- and imputation-approach created on the the training to the test data) is the correct way to assess the performance of the approach. You are right that doing this before the train-test split (and even doing it before the train-validation split for each fold for cross-validation) is potentially problematic. As things go, this is a much less bad leakage between training and test data than other things one could do, but should be avoided as a matter of principle. In this case, I would be hard-pressed to figure out exactly which effect not doing it has, it's so much easier to do this after splitting than trying to figure out what it does (and this only becomes more critical, once you do imputation, feature building or encoding that somehow makes use of the prediction target).

By the way, it seems common to also add a 0/1 variable for "is this variable X imputed" (i.e. one extra variable for every variable that could be missing).

Is this the best method for imputing missing data/and or one that will generalize well to new data? That depends... E.g. if the missingness has predictive value in itself that should be used (then having a mean value + a flag could be decent: example when people do not have a previous purchase history with your online shop [=missing previous purchase information], they might plausibly be riskier customers in some sense than those with a long history) vs. when it should not be used (e.g. missingness is related to the outcome, because the predicted outcome causes the missingness, but you cannot really use this in practice to predict the outcome because of the temporal order of events - e.g. you try to predict death and you measure something on patients and they need to be alive for the measurement, but some die before you get your measurement; it could also be really bad from a discrimination perspective where some underserved communities might have missing data) vs. the missingness in itself is not informative, but one could impute quite well (then something like multiple imputation could work well or perhaps missForest or the like). There's all sorts of variations of this.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.