12
$\begingroup$

I have been working on a binary prediction problem using logistic regression. Using a selection of categorical and continuous variables, I have been able to predict accuracy on a testing set with an AUC of about $0.7$.

I have been comparing multiple data pre-processing approaches where I carry out combinations of various filtering steps which are:

  • no data filtering
  • removing mean based outliers without replacement
  • removing mean based outliers with mean replacement & additionally replacing NA's with the mean.
  • removing median absolute deviation outliers without replacement
  • removing median absolute deviation outliers with mean replacement & additionally replacing NA's with the mean.
  • repeating the above 5 procedures on a data set that has all of the NA's removed.

I find that my model is the most predictive on a testing set whenever I remove all median absolute deviation outliers and replace them with the mean and additionally replace pre-existing NA's with the mean.

Is it OK to impute mean based missing values with the mean whenever implementing the model?

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
4
$\begingroup$

Yes.

It is fine to perform mean imputation, however, make sure to calculate the mean (or any other metrics) only on the train data to avoid data leakage to your test set.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Many thanks for your response. However, wouldn't the use of the training mean to impute for both/either or missing values and and outliers on the testing set be a kind of data leakage to the test set? Then the model would of course perform better than it should because there is data in the testing set that is based upon training data (the same data used to create said model)? $\endgroup$
    – pd441
    Commented Sep 5, 2017 at 7:38
  • $\begingroup$ All your training data shoud be available during the test phase (and in fact is implicitly used in your model). Just note that in case of k-fold CV the train and test might switch roles numerous times $\endgroup$
    – Dimgold
    Commented Sep 5, 2017 at 15:20
3
$\begingroup$

Is it ok to impute mean based missing values with the mean whenever implementing the model?

Yes, as long as you use the mean of your training set---not the mean of the testing set---to impute. Likewise, if you remove values above some threshold in the test case, make sure that the threshold is derived from the training and not test set.

You might also consider holding out two "test" sets and trying all of the methods described above on one of them (using this set to "select" a method) and using the second to estimate error of the method that works best (using this set to "evaluate" the selected method). You would then have a train-validation-test split, which is good practice.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ This is copied and pasted from a comment I made above above however it applies to your answer and I would be interested in a response.... wouldn't the use of the training mean to impute for both/either or missing values and and outliers on the testing set be a kind of data leakage to the test set? Then the model would of course perform better than it should because there is data in the testing set that is based upon training data (the same data used to create said model)? $\endgroup$
    – pd441
    Commented Sep 5, 2017 at 7:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.