Implementing linear regression with standardization

Question

I have this confusion related to implementing linear regression with normalization. Let's say I have a training set trainX and trainY, and test set testX and testY. For the training set I take the mean and standard deviation of trainX, use it to transform the trainX data to have a mean center and unit standard deviation. I do the same for trainY. Now I run a ridge regression for training. For cross validation, I use 10 fold and then get some coefficients optimal.

Now when I use these coefficients on the test sets testX and testY, I need to mean center and give unit standard deviation to both testX and testY using the mean and standard deviation I got from training data sets. I apply the coefficients to predict Y. To these predicted Y values, do I again need to add the mean and standard deviation used before to get the actual Y values? Is this the way to go?

Answer 1

Writing down exactly what you're doing almost always helps figure these things out.

Let the cases in the training set have independent values $x_i$ (which can be considered vectors in a multiple regression setting) and dependent values $y_i$, $1 \le i \le n$. For the test set let's just append primes to the symbols, giving independent values $x'_i$ and dependent values $y'_i$, $1\le i \le n'$. Use a parallel notation for means and standard deviations: the means of the $x_i$ are $\mu$ (a vector), the mean of the $y_i$ is $\nu$; the standard deviations of the $x_i$ are $\sigma$ (also a vector) and the SD of the $y_i$ is $\tau$. The standardized values are $(x_i - \mu)/\sigma$ and $(y_i - \nu)/\tau$.

The model with its fitted parameters $\hat\theta$, in terms of these standardized values, is something of the form

$$\frac{y_i - \nu}{\tau} = f\left(\frac{x_i-\mu}{\sigma}; \hat\theta, \varepsilon\right).$$

(The random variables $\varepsilon$ are there in order to make sense of the equalities, for ordinarily the left hand side does not exactly equal the right hand side.)

Equivalently, by clearing the fraction on the left,

$$y_i = \nu + \tau f\left(\frac{x_i-\mu}{\sigma}; \hat\theta, \varepsilon \right).$$

This is what you want to test, so you will plug in the testing values $x'_i$ and compare the predictions to the $y'_i$ (for vanishing values of the random "errors" $\varepsilon$):

$$y'_i\quad \text{vs}\quad \nu + \tau f\left(\frac{x'_i-\mu}{\sigma}; \hat\theta, 0 \right).$$

The means and SDs of the testing dataset play no role: all standardization continues to be performed using the means and SDs of the training dataset, because they were entered into the model as constants at the outset.