4
$\begingroup$

Lets say we are looking to estimate the following standard OLS regression:

$y_{i} = \beta_{0} + \beta_{1}*X_{i} + \beta_{2}*Z_{i} + \epsilon_{i}$

and that we choose to standardize $X$ as: $X^{*}=((X_{i} - \mu)/ \sigma)$. Suppose we now instead estimate the following regression:

$y_{i} = \beta_{0} + \beta^{*}_{1}*X^{*}_{i} + \beta_{2}*Z_{i} + \epsilon_{i}$

While I know that the point estimates for $\beta_{1}$ and $\beta^{*}_{1}$ will be different, I am less certain of how the significance of the point estimate will be affected. My guess is that a simple transformation of an independent variable (or standardization) would not affect the significance of the point estimate (i.e. it would not affect the t-statistic for its $\beta$ estimate)--but I am not sure of this...

My Question: Is my guess correct? An explanation of why this guess might be right/wrong would be much appreciated.

Thanks!

PS: Running some simple simulations verified my guess above, but a simple simulation is obviously no proof, so if anyone could provide one that would be great.

$\endgroup$
1

1 Answer 1

3
$\begingroup$

Yes, the guess is correct and here is an example code which also proves the same :)

Load the data

> library(MASS)

Here is the ordinary model

> print(summary(lm(medv~crim+rm+age, data=Boston)))

Call:
lm(formula = medv ~ crim + rm + age, data = Boston)

Residuals:
    Min      1Q  Median      3Q     Max 
-19.959  -3.143  -0.633   2.150  39.940 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) -23.60556    2.76938  -8.524  < 2e-16 ***
crim         -0.21102    0.03407  -6.195 1.22e-09 ***
rm            8.03284    0.40201  19.982  < 2e-16 ***
age          -0.05224    0.01046  -4.993 8.21e-07 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 6.094 on 502 degrees of freedom
Multiple R-squared:  0.5636,    Adjusted R-squared:  0.561 
F-statistic: 216.1 on 3 and 502 DF,  p-value: < 2.2e-16

Here is the scaled version

> print(summary(lm(scale(medv)~scale(crim)+scale(rm)+scale(age), data=Boston)))

Call:
lm(formula = scale(medv) ~ scale(crim) + scale(rm) + scale(age), 
data = Boston)

Residuals:
    Min      1Q  Median      3Q     Max 
-2.1701 -0.3417 -0.0688  0.2338  4.3427 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept) -3.077e-16  2.945e-02   0.000        1    
scale(crim) -1.974e-01  3.186e-02  -6.195 1.22e-09 ***
scale(rm)    6.137e-01  3.071e-02  19.982  < 2e-16 ***
scale(age)  -1.599e-01  3.202e-02  -4.993 8.21e-07 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.6626 on 502 degrees of freedom
Multiple R-squared:  0.5636,    Adjusted R-squared:  0.561 
F-statistic: 216.1 on 3 and 502 DF,  p-value: < 2.2e-16

Pretty much everything except the coefficients and residual std. errors remained the same

I don't know the exact answer but intuition goes like this: those t statistics are essentially measures indicating whether the slope is different than 0 (i.e., whether the relation exist). They are calculated as $t=\frac{\beta}{SE_\beta}$. If the relationship is valid, then it is going to be high enough to be considered significant (given the number of observations). When we rescale the data, we will change the value for $\beta$, but at the same time change the $SE_\beta$.

If we resale the whole thing, then the variability which is explained is still there and not changed. if we do any type of manipulation of the predictors (scale all of them, scale single predictor, scale just outcome variable), the same variability is there, it can be explained in the same amount as it is about the relationship between the variables.

Imagine one weird example. You're fitting a regression model HeightInMeters ~ HeightInInches

The t statistic is infitite, SE = 0, the model is perfect, $R^2=1.0$ and so on...

What would change if we change now the predictor so instead of being in inches, it is in yards? It would not matter, the relationship is of the same strength and is able to explain the same amount of variability in the outcome variable. Now, how this translates to math formulas I just don't know :)

$\endgroup$
2
  • 1
    $\begingroup$ Thanks @Vitomir. I like the intuitive arguments that you point out, and your approach is similar to how I approached the problem, but I was looking for more of a formal proof. E.g. a condition which proves that after performing transformation $g$ onto $\beta_{1}$ in the first regression, then $t_{b} = \beta_{1} / SE_{b} = g(\beta_{1}) / SE_{g(b)} = t_{g(b)}$. Best! $\endgroup$
    – Seb
    Apr 12, 2014 at 1:43
  • $\begingroup$ For a formal proof, use the matrix formulation, write out standardization as a matrix multiplication $\endgroup$ Jul 26, 2021 at 0:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.