Standardization and test prediction in linear regression

Question

I am considering a linear regression model to predict a target $y\in\mathbb{R}^n$ from a data matrix $X\in\mathbb{R}^{n\times d}$. Let $X_1,\ldots,X_d\in\mathbb{R}^n$ be the columns of $X$. The prediction of $y$ is $\hat{\alpha}1_n+X\hat{\beta}$, where $$\hat{\alpha},\hat{\beta}\in\arg\min_{\alpha\in\mathbb{R}\\\beta\in\mathbb{R}^d}\|y-\alpha 1_n-X\beta\|_2^2.$$

1. It is common practice to standardize the columns of $X$ to obtain a new matrix $X'$, for, e.g., interpretability: the columns of $X'$ are, for all $j\in\{1,\ldots,d\}$, $$X_j'=\frac{1}{\sigma_j}(X_j-\mu_j1_n),\quad\text{where}\quad\mu_j=\frac{1}{n}\sum_{i=1}^nX_{i,j}\quad\text{and}\quad\sigma_j=\frac{1}{n}\sum_{i=1}^nX_{i,j}^2-\mu_j^2.$$ Then the prediction of $y$ is $\hat{\alpha}'1_n+X'\hat{\beta}'$, where $$\hat{\alpha}',\hat{\beta}'\in\arg\min_{\alpha\in\mathbb{R}\\\beta\in\mathbb{R}^d}\|y-\alpha 1_n-X'\beta\|_2^2.$$ While I understand that this does not affect the prediction accuracy of $y$, what about the prediction of a test response? Given a test data matrix $X_\text{test}\in\mathbb{R}^{m\times d}$, the predicted response is $\hat{\alpha}'1_m+X_\text{test}'\hat{\beta}'$, where $X_{\text{test},j}'=(1/\sigma_j)(X_{\text{test},j}-\mu_j1_m)$. Without standardization, the predicted response is $\hat{\alpha}1_m+X_\text{test}\hat{\beta}$, which seems more direct and probably more accurate. Are there situations where we are still better off using standardization for test prediction?

2. Suppose I also standardize $y$ to obtain a new target $y'$, which is not necessarily useful (or is it?) but not incorrect either. Then the prediction of $y'$ is $\hat{\alpha}''1_n+X'\hat{\beta}''$, where $$\hat{\alpha}'',\hat{\beta}''\in\arg\min_{\alpha\in\mathbb{R}\\\beta\in\mathbb{R}^d}\|y'-\alpha 1_n-X'\beta\|_2^2,$$ and the prediction of $y$ is $\sigma_y(\hat{\alpha}''1_n+X'\hat{\beta}'')+\mu_y1_n$. By the way, we know that $\hat{\alpha}''=0$ (Why does the y-intercept of a linear model disappear when I standardize variables?). Is the test prediction $\sigma_y(\hat{\alpha}''1_m+X_\text{test}'\hat{\beta}'')+\mu_y1_m$? How does its accuracy compare to that in 1.?

Thanks!

Answer 1

The big rule for these transformations is that you learn them from the test set and then apply those to the out-of-sample data. If your standardization of in-sample $x$ involves subtracting a number from every value and then dividing by another number, use those same number when you standardize out-of-sample $x$. Do not calculate those numbers from the out-of-sample data.

Ditto for $y$.

If you do something to the training data, you must do that same transformation to the testing data! If you get coefficient estimates for standardized data and then test with unstandardized data, those coefficient estimates apply for standardized inputs, not unstandardized inputs.

Answer 2

Without loss of generality, a simple linear regression is assumed. Write the sample mean of the regressor as $\overline{x} = \frac{1}{n}X^\top 1_n$, and its standard deviation as $s$. Then, the centered and scaled regressor is $X^* = (X - \overline{x})/s$. Now, observe the relationship between the regression models with nonstandardized and standardized regressors. The nonstandardized one is $$ Y = \alpha + \beta X + \epsilon, $$ whereas with the standardized covariate vector, it becomes $$ Y = \alpha^* + \beta^* X^* + \epsilon = \alpha^* + \beta^* \frac{X - \overline{x}}{s} + \epsilon = \left(\alpha^* - \frac{\beta^* \overline{x}}{s} \right) + \frac{\beta^*}{s}X + \epsilon. $$ We see that $\alpha = \alpha^* - \frac{\beta^*}{s} \overline{x}$ and $\beta = \beta^*/s$. Since $\widehat{\alpha} = \overline{y}-\widehat{\beta}\overline{x}$, $\widehat{\alpha}^* = \overline{y}$ and $\widehat{\beta}^* = s\widehat{\beta}$. This shows you that standardizing the regressor doesn't really change the problem much.

But if you look at the covariance between $\widehat{\alpha}$ and $\widehat{\beta}$, this has a greater implication than just interpretability: $\mathrm{Cov}(\widehat{\alpha},\widehat{\beta}) = -\frac{\overline{x}}{s}$. If the regressor is standardized, $\overline{x}^* = 0$, and thus, $\mathrm{Cov}(\widehat{\alpha}^*,\widehat{\beta}^*) = 0$. This is often very important for accelerating estimation, but I digress. Just note that this is conducive to inference, rather than prediction.

Now, let's say you have a new regressor value $x_\text{new}$. It should be standardized using the same $\overline{x}$ and $s$ as before: $x_\text{new}^* = (x_\text{new}-\overline{x})/s$. Observe that the prediction with this standardized predictor is equivalent to using the unstandardized linear regression and unstandardized regression coefficient estimates. $$ y_\text{new}^* = \widehat{\alpha}^* + \widehat{\beta}^*x_\text{new}^* = \widehat{\alpha}^* + \widehat{\beta}^*\frac{x_\text{new}-\overline{x}}{s} = \left(\widehat{\alpha}^* - \frac{\beta^*\overline{x}}{s} \right) + \frac{\widehat{\beta}^*}{s}x_\text{new}=\widehat{\alpha}+\widehat{\beta}x_\text{new}. $$

The final takeaway is that as long as you transform predictors correctly, the prediction value does not change.