0
$\begingroup$

Suppose I run a multiple regression on $p$ predictors with a sample size of $n$. The degrees of freedom of the regression are then $\mathrm{df} = n - p - 1$, with the extra $-1$ coming from the intercept.

But suppose now that for the same choice of predictors, I choose to standardise my data. It is a well-known result that the intercept vanishes for a multiple regression carried out in this case; does this however imply that $\mathrm{df} = n - k$? Something about this makes me uneasy: given the symmetry of the setup, and the possibility of transforming back and forth, I would have liked to have the $\mathrm{df}$'s agree, but I cannot account for the $\mathrm{-1}$ in the second case. Could it be that in standardising the variables I "use up" one degree of freedom and therefore still have $\mathrm{df} = n - p - 1$ even though this second regression involves no intercept?

$\endgroup$
2
  • $\begingroup$ It is a well-known result that the intercept vanishes for a multiple regression carried out in this case wat? $\endgroup$ Jul 7 '21 at 8:37
  • $\begingroup$ @user2974951 stats.stackexchange.com/q/137103/277115 . Or did you take issue with my phrasing? $\endgroup$
    – Anthony
    Jul 7 '21 at 8:49
1
$\begingroup$

If you standardize all your variables (independent and dependent) then you do not need to include an intercept, because then the intercept will be 0, the standardization made sure of it. In this case the $DF$ will be $n-k$ because you are estimating one parameter less.

This is not a "cheat". You are actually missing one piece of information which you do not have in the original model... the intercept. Yes the intercept is 0, but what does that translate to on the original scale?

If you want to find what the effect of a standardized variable is on the original scale you can unstandardize it. There is no such option for the intercept as it is missing in your model (you are missing a constant - an average). Therefore, if you want to unstandardize the intercept you will have to include it into the model, which will add that missing $DF$.

By removing the intercept you are essentially saying that you do not care for the intercept, not on the original or the standardized scale. If this information is of interest to you then you will have to include it, and the $DF$ will be the same.

Note: if you only standardize your independent variables then you still have to include the intercept (if of interest).

$\endgroup$
6
  • $\begingroup$ Thank you for the answer. A (for me) somewhat unexpected consequence of your argument is that if I convert the $\beta$'s to $b$'s (and thereby move to a model with intercept), since the $\mathrm{df}$ reduces by 1, the $\beta$'s and $b$'s would have slightly different $p$-values. Is this correct? $\endgroup$
    – Anthony
    Jul 7 '21 at 9:57
  • $\begingroup$ @Anthony Correct. Compared to the model without an intercept, the model with an intercept should have slightly higher standard errors, which will result in slightly higher p-values. $\endgroup$ Jul 7 '21 at 10:18
  • $\begingroup$ A small quibble with your comment: the $t$-statistic of $\beta_i$ and $b_i$ are identical since each $\beta_i$ and its standard error are scaled by the same quantity, namely $\sigma(y)/\sigma(x_i)$ where $\sigma(\cdot)$ is the standard deviation. Therefore any difference in the $p$-value between these two quantities would be solely due to the difference in $\mathrm{df}$ employed for the $t$-test. I find this problematic: how can it be that changing the scale of the model (via standardisation) leads to a different significance level for the same effect? $\endgroup$
    – Anthony
    Jul 7 '21 at 10:48
  • 1
    $\begingroup$ @Anthony No, standardization does not change the p-values, the inclusion/exclusion of the intercept changes them. The p-values will be different between a model with an intercept and one without, whether the variables are scaled or not. Maybe I did not understand what you were referring to by $\beta$ and $b$. $\endgroup$ Jul 7 '21 at 10:53
  • $\begingroup$ Thank you, this is what I was getting at. To rephrase for my understanding, "the $p$-value of an effect depends on the model in which the effect has been calculated; a model with an intercept is a different one from a model without one, therefore it is to be expected that the $p$-values differ between these models." $\endgroup$
    – Anthony
    Jul 7 '21 at 11:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.