My workflow is to first split the whole dataset into a train and test set. Then I apply cross validation on the training set. When using cross validation it is important to do standarization repetedly on each training set and then apply the same parameters on the validation part.
My question is how to standardize the original test set? Wich of all the possible training data parameters used for standardization should I use for this test set?
We need to revisit the definition of k-fold cross-validation: train set is divided into k equal parts. Each part is used for validation once. Cross-validation error is the average of k validation errors:
$CV(\lambda) = \frac{1}{k}\sum\limits_{i=1}^{k}E_i(\lambda)$
(For more details on cross-validation, read Hastie & Tibshirani, 2009)
As such, the standardization parameters (mean and standard deviation) for the test set should be the "average" of those used during cross-validation.
Let $\mu_i$'s $(i=1,2,...,k)$ be the means used during cross-validation. Then, the "average" mean used for standardizing the test set is:
which is mean of the train set.
It's tempting (and intuitive) to extrapolate this result and use the standard deviation of the whole train set to standardize the test set. However, the algebra for deriving $\sigma_{avg}$ is trickier so I haven't been able to prove it. Maybe someone can help me?
The main objetive is the standardized data should have the same sense in all the dataset. Then, the normalization should be done over the training sets, and apply these means and SDs over the test set. After, you run the CV in the normalized training set.
