2
$\begingroup$

I want to make predictions using several supervised Machine Learning algorithms and apply 10-fold-cross validation. For doing so, I randomly divided my dataset into in-sample and out-of-sample sets. Moreover, I randomly subdivided the in-sample dataset into training and test set for being able to perform cross-validation. I want to perform a regression task and the predictors include both, numerical and categorical predictors.

As I am quite new to the field of Machine Learning, I am not sure on which subsets I need to perform the following data preprocessing steps.

  • K-nearest-neighbor imputation of missing numerical data
  • Outlier elimination
  • Log transforming variables with a right skew (on which subset do I need to determine the skew?)
  • Feature selection using filter methods (This includes determining the Pearson correlation or Chi-square of predictors with other predictors and with the target variable as well as determining the variance of predictors)
  • Further dataset descriptions such as histograms, mean, median, variance, ...

I am aware that I am trying to prevent a look-ahead bias. However, I am not sure whether I need to perform these steps only on the training subset? Or on the entire in-sample set? Or some of them even on the entire dataset?

Improve this question
$\endgroup$
3
  • $\begingroup$ Consider that the test set should be used in testing only in order to give you an idea as close as possible of the reality. Cleaning fit, feature selection or other transformation using historical data need to use the training set only. $\endgroup$
    – Mayeul sgc
    Commented Jul 19, 2022 at 9:24
  • $\begingroup$ whenever you perform any analysis the first step is to divide the data into in-sample and out-sample, so that you does not add any bias in the test dataset from the training $\endgroup$
    – micro5
    Commented Jul 19, 2022 at 10:01
  • $\begingroup$ @Mayeulsgc But how can I do these things with 10 different training sets? See my question in the answer by frank. $\endgroup$
    – moby1209
    Commented Jul 19, 2022 at 10:21

1 Answer 1

Reset to default
1
$\begingroup$

You have to make sure that your preprocessing of the training set doesn't use information from the validation or test set ("leakage"). E.g., in your k-nearest neighbor imputation, you must not include data from the validation or test set in this k-NN method.

Similarly for the outlier elimination, transformation, feature selection, ... All of that could potentially leak information from the validation or test set into the model.

Note that leakage can sometimes be very subtle. So, to be on the save side, repeat all those preprocessing anew with each training set.

Improve this answer
$\endgroup$
10
  • $\begingroup$ That sounds reasonable. However, in my paper, I also need to report statistics such as histograms, etc. But this is a problem when I have 10 different test sets. Is there a common solution to this? $\endgroup$
    – moby1209
    Commented Jul 19, 2022 at 10:18
  • $\begingroup$ Moreover, how can I then make decisions regarding log transformations, feature selection, etc. when I have 10 different training sets? $\endgroup$
    – moby1209
    Commented Jul 19, 2022 at 10:18
  • 1
    $\begingroup$ @moby1209 The ten-fold cross-validation is only for finding the best model. Once you have it, you throw away all the inferior models. Then you take the complete in-sample dataset and apply again preprocessing and training (and finally test it on the single test set). $\endgroup$
    – frank
    Commented Jul 19, 2022 at 10:32
  • 1
    $\begingroup$ I don't understand: If you cannot train without leakage, then either don't do it or live with the knowledge that your results might not generalize well. $\endgroup$
    – frank
    Commented Jul 19, 2022 at 10:59
  • 1
    $\begingroup$ Yes. Unless you are sure that it does not leak information into the training set, which is often hard to tell. $\endgroup$
    – frank
    Commented Jul 19, 2022 at 11:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.