4
$\begingroup$

With reference to this post on feature scaling, and many tutorials out there, it is mentioned how we should avoid data snooping by performing say, feature scaling on the train set, and then use the mean and std from the train set to transform the test set.

I can understand this idea, but when extending to say, KFold cross validation (K=5), how do we then determine the mean and std for our final test set?

I was thinking the below:

  1. Split X, y into 3 sets, X_train, X_test (note no validation set here since we will be splitting X_train into 5 folds).
  2. X_train is further split into X1,X2,X3,X4,X5 (for simplicity, I dropped the train subfix.
  3. We will train the model five times, for example, use X2 - X5 as train and evaluate on X1. Use X1-X4 as train, and evaluate on X5.
  4. The confusion arises here, during training for each fold, we should perform feature scaling on only 4 folds (the training folds), and use the mean and std from these 4 folds to transform the remaining 1 validation fold. But this will mean we have 5 such pairs of mean - std since CV=5.
  5. How do we deduce which mean and std to use for the final final prediction on test set? Logically, we just choose the mean and std of the best performing fold (out of 5 folds?) This leads to the final question.
  6. Don't people typically take the average of the 5 fold predictions at the end when doing inference, so do we still take the best performing folds' mean and std?
Improve this question
$\endgroup$

2 Answers 2

Reset to default
5
$\begingroup$

In the presence of a test set, the Kfold is tpically done for hyper parameter optimization, and after that, you should train your model with the whole training set and estimate the mean/std from it to be used on the test set.

Improve this answer
$\endgroup$
7
  • 2
    $\begingroup$ Thanks. Oh this clears up the air. Thanks so much. So this is a bit different from deep learning (say image recognition) where you train five folds and use five weights to perform inference and average the predictions. Here you are using cross validation for Hyperparameter tuning and after you choose the Hyperparameter, you use this set of “ideal” Hyperparameter to retrain your whole train set (where this time you use the mean and std of the whole train set). $\endgroup$
    – nan
    Commented Sep 24, 2021 at 14:29
  • 2
    $\begingroup$ @nan That is right. And, this is model agnostic. It can be applied to deep learning as well, and also the method you described can be applied to algorithms other than DL. $\endgroup$
    – gunes
    Commented Sep 24, 2021 at 14:32
  • 1
    $\begingroup$ Yes, this is called meta-estimation. $\endgroup$
    – gunes
    Commented Sep 24, 2021 at 14:47
  • 1
    $\begingroup$ Roger that @gunes. Thanks a lot, I understood the idea now. That’s why people like me should never jump the gun and learn deep learning without a solid foundation in classical ML. $\endgroup$
    – nan
    Commented Sep 24, 2021 at 14:48
  • 2
    $\begingroup$ You’ll use each model’s own mean/std just as you use each model’s own weights. $\endgroup$
    – gunes
    Commented Sep 24, 2021 at 15:14
3
$\begingroup$

Everything you do to the data is part of your model. This includes standardization.

In each fold, you need to pretend that the fold is your only training set. This means that for 5 fold cross validation, you would learn a new mean and standard deviation and apply that to the hold out set prior to predicting.

Sklearn makes this very easy by using sklearn.pipeline.Pipe and sklearn.preprocessing.StandardScaler.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for the fast answer. So we learn 5 pairs of mean and std during five folds cv. However, in the final test set, which pair of mean and std do we choose? $\endgroup$
    – nan
    Commented Sep 24, 2021 at 14:26
  • 1
    $\begingroup$ @nan Cross validation is meant to estimate generalization error. You learn the final mean, std when you train on the full training set. $\endgroup$
    – Demetri Pananos
    Commented Sep 24, 2021 at 14:34
  • 3
    $\begingroup$ +1 Always remember that any out-of-sample testing mimics the real application where you want something like Siri doing speech recognition on sentences that have yet to be spoken (perhaps by people who have yet to be born). You literally pretend that all out-of-sample data do not exist, because in the real application, such data really might not yet exist! $\endgroup$
    – Dave
    Commented Sep 24, 2021 at 15:03
  • $\begingroup$ @Dave thanks. In a typical machine learning project, can I treat the error on the test set to be the out of sample error? $\endgroup$
    – nan
    Commented Sep 24, 2021 at 15:34
  • 2
    $\begingroup$ @nan Once you finalize the model you're going to use, then yes the error on the test set is the estimate of the generalization error assuming the data generating processes stays fixed $\endgroup$
    – Demetri Pananos
    Commented Sep 24, 2021 at 15:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.