A common good practice in Machine Learning is to do feature normalization or data standardization of the predictor variables, that's it, center the data substracting the mean and normalize it dividing by the variance (or standard deviation too). For self containment and to my understanding we do this to achieve two main things:
- Avoid extra small model weights for the purpose of numerical stability.
- Ensure quick convergence of optimization algorithms like e.g. Conjugate Gradient so that the large magnitude of one predictor dimension w.r.t. the others doesn't lead to slow convergence.
We usually split the data into training, validation and testing sets. In the literature we usually see that to do feature normalization they take the mean and variance (or standard deviation) over the whole set of predictor variables. The big flaw I see here is that if you do that, you are in fact introducing future information into the training predictor variables namely the future information contained in the mean and variance.
Therefore, I do feature normalization over the training data and save the mean and variance. Then I apply feature normalization to the predictor variables of the validation and test data sets using the training mean and variances. Are there any fundamental flaws with this? can anyone recommend a better alternative?