Normalization prior to cross-validation

Question

Does normalizing data (to have zero mean and unity standard deviation) prior to performing a repeated k-fold cross-validation have any negative conquences such as overfitting?

Note: this is for a situation where #cases > total #features

I am transforming some of my data using a log transform, then normalizing all data as above. I am then performing feature selection. Next I apply the selected features and normalized data to a repeated 10-fold cross-validation to try and estimate generalized classifier performance and am concerned that using all data to normalize may not be appropriate. Should I normalize the test data for each fold using normalizing data obtained from the training data for that fold?

Any opinions gratefully received! Apologies if this question seems obvious.

Edit: On testing this (in line with suggestions below) I found that normalization prior to CV did not make much difference performance-wise when compared to normalization within CV.

Answer 1

To answer your main question, it would be optimal and more appropiate to scale within the CV. But it will probably not matter much and might not be important in practice at all if your classifier rescales the data, which most do (at least in R).

However, selecting feature before cross validating is a BIG NO and will lead to overfitting, since you will select them based on how they perform on the whole data set. The log-transformation is ok to perform outside, since the transformation does not depend on the actual data (more on the type of data) and is not something you would not do if you had only 90% of the data instead of 100% and is not tweaked according to the data.

To also answer your comment, obviously whether it will result in overfitting will depend on your manner of feature selection. If you choose them by chance (why would you do that?) or because of a priori theoretical considerations (other literature) it won't matter. But if it depends on your data set it will. Elements of Statistical Learnings has a good explanation. You can freely and legally download a .pdf here http://www-stat.stanford.edu/~tibs/ElemStatLearn/

The point concerning you is in section 7.10.2 on page 245 of the fifth printing. It is titled "The Wrong and Right Ways to do Cross-validation".

Answer 2

Cross-validation is best viewed as a method to estimate the performance of a statistical procedure, rather than a statistical model. Thus in order to get an unbiased performance estimate, you need to repeat every element of that procedure separately in each fold of the cross-validation, which would include normalisation. So I would say normalise in each fold.

The only time this would not be necessary is if the statistical procedure was completely insensitive to the scaling and mean value of the data.

Answer 3

I think that if the normalization only involves two parameters and you have a good size sample that will not be a problem. I would be more concerned about the transformation and the variable selection process. 10 fold cross-validation seems to be the rage today. Doesn't anybody use bootstrap 632 or 632+ for classifier error rate estimation as suggested first by Efron (1983) in JASA and followed-up later in a paper by Efron and Tibshirani with the 632+?