Follow-up question: When should you center your data & when should you standardize?

Question

I have a follow up question to MånsT's reply to the "When should you center your data & when should you standardize"-question. ( I cannot leave a comment as I am below the magic "50 reputation".) He says that

In addition to the remarks in the other answers, I'd like to point out that the scale and location of the explanatory variables does not affect the validity of the regression model in any way.

Consider the model $y=\beta_0+\beta_1x_1+\beta_2x_2+…+ϵ $.

The least squares estimators of $\beta_1,\beta_2$,… are not affected by shifting. The reason is that these are the slopes of the fitting surface - how much the surface changes if you change $x_1,x_2$,… one unit. This does not depend on location. (The estimator of $\beta_0$, however, does.)

By looking at the equations for the estimators you can see that scaling $x_1$ with a factor $a$ scales $\beta_1$ by a factor 1/$a$. To see this, note that

$\hat{\beta_1}(x_1)=\frac{\sum_{i=1}^n (x_{1,i}−\bar{x_1})(y_i−\bar{y})}{\sum_{i=1}^n(x_{1,i}-\bar{x_1})^2}.$

Thus

$$ \hat{\beta_1} (ax_1)=\frac{\sum_{i=1}^n (ax_{1,i}−ax_1)(y_i−\bar{y})}{\sum_{i=1}^2(ax_{1,i}−a\bar{x_1})^2}=\frac{a\sum_{i=1}^n (x_{1,i}−\bar{x_1})(y_i−\bar{y})}{a^2\sum_{i=1}^n(x_{1,i}-\bar{x_1})^2}=\frac{\hat{\beta_1}(x_1)}{a}.$$

By looking at the corresponding formula for $\beta_2$ (for instance) it is (hopefully) clear that this scaling doesn't affect the estimators of the other slopes.

It is clear to me that $\hat{\beta_1}$ is given by the provided expression above. However, this is the $\beta_1$ in the model: $y=\beta_0+\beta_1x.$, as MånsT has clarified by writing $\hat{\beta_1} (x_1)$. Will this argument still hold in the general linear model, given by $y=\beta_0+\beta_1x_1+\beta_2x_2+…+ϵ $?? Since, in the general linear model, the $\beta$s are given by the normal equations: $\vec{\beta}=(X^t X)^{-1}X^t \vec{y}$ (full rank of $X^t X$ assumed).

So: It is not clear to me that the $\beta_1$ in the model with only one predictor will be the same as the corresponding $\beta_1$ in $\vec{\beta}$. According to http://data.princeton.edu/wws509/notes/c2.pdf, pages 21-24, there is a difference (the author makes the distinction between gross and net effects (gross: $\beta_1$ in one predictor model, net: $\beta_1$ in general linear model)). How will the scale-invariance argument be for the general linear model?

Answer 1

Suppose a model of the form

$$\mathbb{E}(Y) = f(\beta_0 + \beta_1 x_1 + \cdots + \beta_d x_d)$$

with parameters $(\beta_0, \beta_1, \ldots, \beta_d)$ is fit to data $((x_{i1}, x_{i2}, \ldots, x_{id}, y_i))$ by optimizing some measure of discrepancy between the $y_i$ and the corresponding $\mathrm{x}_i$, as is done (for instance) with least squares, generalized least squares, and maximum likelihood estimators for generalized linear models. Further suppose the optimum is unique. Write its value as $\hat{\beta} = (\hat{\beta}_0, \hat{\beta}_1, \ldots, \hat{\beta}_d)$. Then, for any nonzero constants $\sigma_1, \ldots, \sigma_d$ and any constants $\mu_1, \ldots, \mu_d$, it is immediate (by plugging in the values) that

$$\hat{\alpha}_0 = \hat{\beta}_0 - \hat{\beta}_1\mu_1/\sigma_1 - \cdots - \hat{\beta}_d\mu_d\sigma_d$$

and

$$(\hat{\alpha}_1, \ldots, \hat{\alpha}_d) = (\hat{\beta}_1/\sigma_1, \ldots, \hat{\beta}_d/\sigma_d)$$

is the unique optimum for the model

$$\mathbb{E}(Y) = f(\alpha_0 +\alpha_1 z_1 + \cdots + \alpha_d z_d)$$

with parameters $(\alpha_1, \ldots, \alpha_d)$ and new (affinely) transformed variables

$$z_i = \sigma_i x_i + \mu_i.$$

Even when the optimum is not unique, any optimum for the first model is still converted to some optimum for the second model. It just might not be the same optimum returned by a computer program. This can happen when the parameters are not identifiable, for instance.

When the constant term $\beta_0$ is not included in the model, it is no longer possible to carry out this reasoning unless all the $\mu_i=0$. Even then, the estimated coefficients $\hat{\beta}_i$ still transform contravariantly with respect to rescaling of the variables $x_i$.