Perform feature normalization before or within model validation?

Question

A common good practice in Machine Learning is to do feature normalization or data standardization of the predictor variables, that's it, center the data substracting the mean and normalize it dividing by the variance (or standard deviation too). For self containment and to my understanding we do this to achieve two main things:

1. Avoid extra small model weights for the purpose of numerical stability.
2. Ensure quick convergence of optimization algorithms like e.g. Conjugate Gradient so that the large magnitude of one predictor dimension w.r.t. the others doesn't lead to slow convergence.

We usually split the data into training, validation and testing sets. In the literature we usually see that to do feature normalization they take the mean and variance (or standard deviation) over the whole set of predictor variables. The big flaw I see here is that if you do that, you are in fact introducing future information into the training predictor variables namely the future information contained in the mean and variance.

Therefore, I do feature normalization over the training data and save the mean and variance. Then I apply feature normalization to the predictor variables of the validation and test data sets using the training mean and variances. Are there any fundamental flaws with this? can anyone recommend a better alternative?

Answer 1

Your approach is entirely correct. Although data transformations are often undervalued as "preprocessing", one cannot emphasize enough that transformations in order to optimize model performance can and should be treated as part of the model building process.

Reasoning: A model shall be applied on unseen data which is in general not available at the time the model is built. The validation process (including data splitting) simulates this. So in order to get a good estimate of the model quality (and generalization power) one needs to restrict the calculation of the normalization parameters (mean and variance) to the training set.

I can only guess why this is not always done in literature. One argument could be, that the calculation of mean and variance is not that sensitive to small data variations (but even this is only true if the basic sample size is large enough and the data is approximately normally distributed without extreme outliers).

Answer 2

Feature normalization is to make different features in the same scale. The scaling speeds up gradient descent by avoiding many extra iterations that are required when one or more features take on much larger values than the rest(Without scaling, the cost function that is visualized will show a great asymmetry).

I think it makes sense that use the mean and var from training set when test data come. Yet if the data size is huge, both training and validation sets can be approximately viewed as normal distribution, thus they roughly share the mean and var.

Answer 3

The methodology you have described is sound as others have said. You should perform the exact same transformation on your test set features as you do on features from your training set.

I think it's worth adding that another reason for feature normalization is to enhance the performance of certain processes that are sensitive to differences to the scale of certain variables. For example principal components analysis (PCA) aims to capture the greatest proportion of variance, and as a result will give more weight to variables that exhibit the largest variance if feature normalization is not performed initially.

Answer 4

Let me illustrate as to why we have to do normalization only on training data set.Say we extract features of different characteristics of apples like length,breath of the bounding box surrounding the apples,color,etc.,Let's say we normalize the feature length on the whole data and we have got mean length as 7 cm and variance as 1 cm.But if we just use train data we may get mean as 6 cm and variance as 1 cm.Which actually mean that there are large apples in the test data set which was actually not learn in the train data set.