22
$\begingroup$

Does normalizing data (to have zero mean and unity standard deviation) prior to performing a repeated k-fold cross-validation have any negative conquences such as overfitting?

Note: this is for a situation where #cases > total #features

I am transforming some of my data using a log transform, then normalizing all data as above. I am then performing feature selection. Next I apply the selected features and normalized data to a repeated 10-fold cross-validation to try and estimate generalized classifier performance and am concerned that using all data to normalize may not be appropriate. Should I normalize the test data for each fold using normalizing data obtained from the training data for that fold?

Any opinions gratefully received! Apologies if this question seems obvious.

Edit: On testing this (in line with suggestions below) I found that normalization prior to CV did not make much difference performance-wise when compared to normalization within CV.

Improve this question
$\endgroup$

4 Answers 4

Reset to default
17
$\begingroup$

To answer your main question, it would be optimal and more appropiate to scale within the CV. But it will probably not matter much and might not be important in practice at all if your classifier rescales the data, which most do (at least in R).

However, selecting feature before cross validating is a BIG NO and will lead to overfitting, since you will select them based on how they perform on the whole data set. The log-transformation is ok to perform outside, since the transformation does not depend on the actual data (more on the type of data) and is not something you would not do if you had only 90% of the data instead of 100% and is not tweaked according to the data.

To also answer your comment, obviously whether it will result in overfitting will depend on your manner of feature selection. If you choose them by chance (why would you do that?) or because of a priori theoretical considerations (other literature) it won't matter. But if it depends on your data set it will. Elements of Statistical Learnings has a good explanation. You can freely and legally download a .pdf here http://www-stat.stanford.edu/~tibs/ElemStatLearn/

The point concerning you is in section 7.10.2 on page 245 of the fifth printing. It is titled "The Wrong and Right Ways to do Cross-validation".

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks - surely if the only results reported are for the performance estimated obtained using the selected model (features) in each fold this cannot result in overfitting? Afterall you are only reporting the generalized performance of a given feature subset. $\endgroup$
    – BGreene
    Commented May 3, 2012 at 15:56
  • $\begingroup$ I expanded my answer to adress your comment. I think the explanation in the link is better than what I can cook up right now. $\endgroup$
    – Erik
    Commented May 3, 2012 at 16:42
  • $\begingroup$ Thank you. This suggests that standard filter based feature selection should not ever be used and that wrapper based feature selection or equivalent used instead. Does this still hold true for the situation where #Cases > #Features? (I have 259 cases, total of 56 features). I would assume the potential for error is less here? $\endgroup$
    – BGreene
    Commented May 3, 2012 at 17:10
8
$\begingroup$

Cross-validation is best viewed as a method to estimate the performance of a statistical procedure, rather than a statistical model. Thus in order to get an unbiased performance estimate, you need to repeat every element of that procedure separately in each fold of the cross-validation, which would include normalisation. So I would say normalise in each fold.

The only time this would not be necessary is if the statistical procedure was completely insensitive to the scaling and mean value of the data.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I think this is a good answer, although it is not rigorous, it gets the point across. I think that the meaning is that if you normalise in each fold, it is guaranteed that the normalisation doesn't bias performance. The way it was written seemed that there was no other way to avoid bias, although Dikran did point out in the last line that there are other ways. If you do normalisation inside CV, it never increases bias, as opposed to doing it outside, which may. The bias may not affect much anyway, as pointed out in the other answer. $\endgroup$
    – Tom Anderson
    Commented Sep 13, 2016 at 5:14
2
$\begingroup$

I think that if the normalization only involves two parameters and you have a good size sample that will not be a problem. I would be more concerned about the transformation and the variable selection process. 10 fold cross-validation seems to be the rage today. Doesn't anybody use bootstrap 632 or 632+ for classifier error rate estimation as suggested first by Efron (1983) in JASA and followed-up later in a paper by Efron and Tibshirani with the 632+?

Improve this answer
$\endgroup$
0
$\begingroup$

I personally like the .632 method. Which is basically boostrapping with replacement. If you do that and remove duplicates you will get 632 entries out of an input set of 1000. Kind of neat.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ The 632 estimator is different from what you say. It actually is a linear combination of two error rate estimators giving weight 0.632 for the bootstrap $e_0$ estimator and 0.368 to the apparent error rate (also called the resubstitution estimate). $\endgroup$
    – Michael R. Chernick
    Commented May 14, 2018 at 19:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.