I have this confusion related to implementing linear regression with normalization. Let's say I have a training set trainX and trainY, and test set testX and testY. For the training set I take the mean and standard deviation of trainX, use it to transform the trainX data to have a mean center and unit standard deviation. I do the same for trainY. Now I run a ridge regression for training. For cross validation, I use 10 fold and then get some coefficients optimal.

Now when I use these coefficients on the test sets testX and testY, I need to mean center and give unit standard deviation to both testX and testY using the mean and standard deviation I got from training data sets. I apply the coefficients to predict Y. To these predicted Y values, do I again need to add the mean and standard deviation used before to get the actual Y values? Is this the way to go?

  • $\begingroup$ "Normalization" typically means changing values to fall within a given range such as $0$ to $1$ or $0$% to $100$%. You describe standardization, which consists of a linear transformation making the mean equal to zero and the standard deviation equal to unity. $\endgroup$ – whuber Aug 23 '13 at 22:11
  • $\begingroup$ It is very unclear why it is worth more than a few seconds of your time to deal with standardization. $\endgroup$ – Frank Harrell Aug 23 '13 at 22:58

Writing down exactly what you're doing almost always helps figure these things out.

Let the cases in the training set have independent values $x_i$ (which can be considered vectors in a multiple regression setting) and dependent values $y_i$, $1 \le i \le n$. For the test set let's just append primes to the symbols, giving independent values $x'_i$ and dependent values $y'_i$, $1\le i \le n'$. Use a parallel notation for means and standard deviations: the means of the $x_i$ are $\mu$ (a vector), the mean of the $y_i$ is $\nu$; the standard deviations of the $x_i$ are $\sigma$ (also a vector) and the SD of the $y_i$ is $\tau$. The standardized values are $(x_i - \mu)/\sigma$ and $(y_i - \nu)/\tau$.

The model with its fitted parameters $\hat\theta$, in terms of these standardized values, is something of the form

$$\frac{y_i - \nu}{\tau} = f\left(\frac{x_i-\mu}{\sigma}; \hat\theta, \varepsilon\right).$$

(The random variables $\varepsilon$ are there in order to make sense of the equalities, for ordinarily the left hand side does not exactly equal the right hand side.)

Equivalently, by clearing the fraction on the left,

$$y_i = \nu + \tau f\left(\frac{x_i-\mu}{\sigma}; \hat\theta, \varepsilon \right).$$

This is what you want to test, so you will plug in the testing values $x'_i$ and compare the predictions to the $y'_i$ (for vanishing values of the random "errors" $\varepsilon$):

$$y'_i\quad \text{vs}\quad \nu + \tau f\left(\frac{x'_i-\mu}{\sigma}; \hat\theta, 0 \right).$$

The means and SDs of the testing dataset play no role: all standardization continues to be performed using the means and SDs of the training dataset, because they were entered into the model as constants at the outset.

  • $\begingroup$ Do I need to do that to the target variables? I mean the normalization to the target variables? Is it necessary for the ys as well. $\endgroup$ – user34790 Aug 24 '13 at 1:19
  • $\begingroup$ What do you mean by "target variables"? $\endgroup$ – whuber Aug 24 '13 at 2:07
  • $\begingroup$ I believe you may be overthinking this: once you have written down the model based on the training set, you simply apply it to the test set, exactly as always. The model is given in my second-to-last formula and the comparison you need to make is in the last formula. The $x'$ and $y'$ literally are plugged right in. $\endgroup$ – whuber Aug 24 '13 at 14:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.