Return to Question

I have a follow up question to MånsT's reply to the "When should you center your data & when should you standardize"-question. ( I cannot leave a comment as I am below the magic "50 reputation".) He says that

In addition to the remarks in the other answers, I'd like to point out that the scale and location of the explanatory variables does not affect the validity of the regression model in any way.

 

Consider the model $y=\beta_0+\beta_1x_1+\beta_2x_2+…+ϵ $.

 

The least squares estimators of $\beta_1,\beta_2$,… are not affected by shifting. The reason is that these are the slopes of the fitting surface - how much the surface changes if you change $x_1,x_2$,… one unit. This does not depend on location. (The estimator of $\beta_0$, however, does.)

 

By looking at the equations for the estimators you can see that scaling $x_1$ with a factor $a$ scales $\beta_1$ by a factor 1/$a$. To see this, note that

 

$\hat{\beta_1}(x_1)=\frac{\sum_{i=1}^n (x_{1,i}−\bar{x_1})(y_i−\bar{y})}{\sum_{i=1}^n(x_{1,i}-\bar{x_1})^2}.$

 

Thus

 

$$ \hat{\beta_1} (ax_1)=\frac{\sum_{i=1}^n (ax_{1,i}−ax_1)(y_i−\bar{y})}{\sum_{i=1}^2(ax_{1,i}−a\bar{x_1})^2}=\frac{a\sum_{i=1}^n (x_{1,i}−\bar{x_1})(y_i−\bar{y})}{a^2\sum_{i=1}^n(x_{1,i}-\bar{x_1})^2}=\frac{\hat{\beta_1}(x_1)}{a}.$$

 

By looking at the corresponding formula for $\beta_2$ (for instance) it is (hopefully) clear that this scaling doesn't affect the estimators of the other slopes.

It is clear to me that $\hat{\beta_1}$ is given by the provided expression above. However, this is the $\beta_1$ in the model: $y=\beta_0+\beta_1x.$, as MånsT has clarified by writing $\hat{\beta_1} (x_1)$. Will this argument still hold in the general linear model, given by $y=\beta_0+\beta_1x_1+\beta_2x_2+…+ϵ $?? Since, in the general linear model, the $\beta$s are given by the normal equations: $\vec{\beta}=(X^t X)^{-1}X^t \vec{y}$ (full rank of $X^t X$ assumed).

So: It is not clear to me that the $\beta_1$ in the model with only one predictor will be the same as the corresponding $\beta_1$ in $\vec{\beta}$. According to http://data.princeton.edu/wws509/notes/c2.pdf, pages 21-24, there is a difference (the author makes the distinction between gross and net effects (gross: $\beta_1$ in one predictor model, net: $\beta_1$ in general linear model)). How will the scale-invariance argument be for the general linear model?

I have a follow up question to MånsT's reply to the "When should you center your data & when should you standardize"-question. ( I cannot leave a comment as I am below the magic "50 reputation".) He says that

In addition to the remarks in the other answers, I'd like to point out that the scale and location of the explanatory variables does not affect the validity of the regression model in any way.

Consider the model $y=\beta_0+\beta_1x_1+\beta_2x_2+…+ϵ $.

The least squares estimators of $\beta_1,\beta_2$,… are not affected by shifting. The reason is that these are the slopes of the fitting surface - how much the surface changes if you change $x_1,x_2$,… one unit. This does not depend on location. (The estimator of $\beta_0$, however, does.)

By looking at the equations for the estimators you can see that scaling $x_1$ with a factor $a$ scales $\beta_1$ by a factor 1/$a$. To see this, note that

$\hat{\beta_1}(x_1)=\frac{\sum_{i=1}^n (x_{1,i}−\bar{x_1})(y_i−\bar{y})}{\sum_{i=1}^n(x_{1,i}-\bar{x_1})^2}.$

Thus

$$ \hat{\beta_1} (ax_1)=\frac{\sum_{i=1}^n (ax_{1,i}−ax_1)(y_i−\bar{y})}{\sum_{i=1}^2(ax_{1,i}−a\bar{x_1})^2}=\frac{a\sum_{i=1}^n (x_{1,i}−\bar{x_1})(y_i−\bar{y})}{a^2\sum_{i=1}^n(x_{1,i}-\bar{x_1})^2}=\frac{\hat{\beta_1}(x_1)}{a}.$$

By looking at the corresponding formula for $\beta_2$ (for instance) it is (hopefully) clear that this scaling doesn't affect the estimators of the other slopes.

It is clear to me that $\hat{\beta_1}$ is given by the provided expression above. However, this is the $\beta_1$ in the model: $y=\beta_0+\beta_1x.$, as MånsT has clarified by writing $\hat{\beta_1} (x_1)$. Will this argument still hold in the general linear model, given by $y=\beta_0+\beta_1x_1+\beta_2x_2+…+ϵ $?? Since, in the general linear model, the $\beta$s are given by the normal equations: $\vec{\beta}=(X^t X)^{-1}X^t \vec{y}$ (full rank of $X^t X$ assumed).

So: It is not clear to me that the $\beta_1$ in the model with only one predictor will be the same as the corresponding $\beta_1$ in $\vec{\beta}$. According to http://data.princeton.edu/wws509/notes/c2.pdf, pages 21-24, there is a difference (the author makes the distinction between gross and net effects (gross: $\beta_1$ in one predictor model, net: $\beta_1$ in general linear model)). How will the scale-invariance argument be for the general linear model?

I have a follow up question to MånsT's reply to the "When should you center your data & when should you standardize"-question. ( I cannot leave a comment as I am below the magic "50 reputation".) He says that

In addition to the remarks in the other answers, I'd like to point out that the scale and location of the explanatory variables does not affect the validity of the regression model in any way.

Consider the model $y=\beta_0+\beta_1x_1+\beta_2x_2+…+ϵ $.

The least squares estimators of $\beta_1,\beta_2$,… are not affected by shifting. The reason is that these are the slopes of the fitting surface - how much the surface changes if you change $x_1,x_2$,… one unit. This does not depend on location. (The estimator of $\beta_0$, however, does.)

By looking at the equations for the estimators you can see that scaling $x_1$ with a factor $a$ scales $\beta_1$ by a factor 1/$a$. To see this, note that

$\hat{\beta_1}(x_1)=\frac{\sum_{i=1}^n (x_{1,i}−\bar{x_1})(y_i−\bar{y})}{\sum_{i=1}^n(x_{1,i}-\bar{x_1})^2}.$

Thus

$$ \hat{\beta_1} (ax_1)=\frac{\sum_{i=1}^n (ax_{1,i}−ax_1)(y_i−\bar{y})}{\sum_{i=1}^2(ax_{1,i}−a\bar{x_1})^2}=\frac{a\sum_{i=1}^n (x_{1,i}−\bar{x_1})(y_i−\bar{y})}{a^2\sum_{i=1}^n(x_{1,i}-\bar{x_1})^2}=\frac{\hat{\beta_1}(x_1)}{a}.$$

By looking at the corresponding formula for $\beta_2$ (for instance) it is (hopefully) clear that this scaling doesn't affect the estimators of the other slopes.

It is clear to me that $\hat{\beta_1}$ is given by the provided expression above. However, this is the $\beta_1$ in the model: $y=\beta_0+\beta_1x.$, as MånsT has clarified by writing $\hat{\beta_1} (x_1)$. Will this argument still hold in the general linear model, given by $y=\beta_0+\beta_1x_1+\beta_2x_2+…+ϵ $?? Since, in the general linear model, the $\beta$s are given by the normal equations: $\vec{\beta}=(X^t X)^{-1}X^t \vec{y}$ (full rank of $X^t X$ assumed).

So: It is not clear to me that the $\beta_1$ in the model with only one predictor will be the same as the corresponding $\beta_1$ in $\vec{\beta}$. According to http://data.princeton.edu/wws509/notes/c2.pdf, pages 21-24, there is a difference (the author makes the distinction between gross and net effects (gross: $\beta_1$ in one predictor model, net: $\beta_1$ in general linear model)). How will the scale-invariance argument be for the general linear model?

I have a follow up question to MånsT's reply to the "When should you center your data & when should you standardize"-question. ( I cannot leave a comment as I am below the magic "50 reputation".) He says that

In addition to the remarks in the other answers, I'd like to point out that the scale and location of the explanatory variables does not affect the validity of the regression model in any way.

Consider the model $y=\beta_0+\beta_1x_1+\beta_2x_2+…+ϵ $.

The least squares estimators of $\beta_1,\beta_2$,… are not affected by shifting. The reason is that these are the slopes of the fitting surface - how much the surface changes if you change $x_1,x_2$,… one unit. This does not depend on location. (The estimator of $\beta_0$, however, does.)

By looking at the equations for the estimators you can see that scaling $x_1$ with a factor $a$ scales $\beta_1$ by a factor 1/$a$. To see this, note that

$\hat{\beta_1}(x_1)=\frac{\sum_{i=1}^n (x_{1,i}−\bar{x_1})(y_i−\bar{y})}{\sum_{i=1}^n(x_{1,i}-\bar{x_1})^2}.$

Thus

$$ \hat{\beta_1} (ax_1)=\frac{\sum_{i=1}^n (ax_{1,i}−ax_1)(y_i−\bar{y})}{\sum_{i=1}^2(ax_{1,i}−a\bar{x_1})^2}=\frac{a\sum_{i=1}^n (x_{1,i}−\bar{x_1})(y_i−\bar{y})}{a^2\sum_{i=1}^n(x_{1,i}-\bar{x_1})^2}=\frac{\hat{\beta_1}(x_1)}{a}.$$

By looking at the corresponding formula for $\beta_2$ (for instance) it is (hopefully) clear that this scaling doesn't affect the estimators of the other slopes.

It is clear to me that $\hat{\beta_1}$ is given by the provided expression above. However, this is the $\beta_1$ in the model: $y=\beta_0+\beta_1x.$, as MånsT has clarified by writing $\hat{\beta_1} (x_1)$. Will this argument still hold in the general linear model, given by $y=\beta_0+\beta_1x_1+\beta_2x_2+…+ϵ $?? Since, in the general linear model, the $\beta$s are given by the normal equations: $\vec{\beta}=(X^t X)^{-1}X^t \vec{y}$ (full rank of $X^t X$ assumed).

So: It is not clear to me that the $\beta_1$ in the model with only one predictor will be the same as the corresponding $\beta_1$ in $\vec{\beta}$. According to http://data.princeton.edu/wws509/notes/c2.pdf, pages 21-24, there is a difference (the author makes the distinction between gross and net effects (gross: $\beta_1$ in one predictor model, net: $\beta_1$ in general linear model)). How will the scale-invariance argument be for the general linear model?