7
$\begingroup$

I understand that in multiple regression analysis, for each independent variable, you would graph dependent variable vs independent variable and you would make a line of best fit and calculate the standard error.

However, how does it work for intercept? Can some one give me a concise but clear explanation?

$\endgroup$

2 Answers 2

4
$\begingroup$

The standard error of the the intercept allows you to test whether or not the estimated intercept is statistically significant from a specified(hypothesized) value ...normally 0.0 . If you test against 0.0 and fail to reject then you can then re-estimate your model without the intercept term being present.

$\endgroup$
2
8
$\begingroup$

Your characterization of how multiple regression works is inaccurate. Your version implies fitting a simple linear regression for each variable in turn (and presumably using each of those slopes as the coefficient for that variable in the multiple regression model). This notion leaves you with the problem of how to deal with the fact that the intercepts from each simple regression are quite likely to differ.

However, that approach is not how multiple regression works / estimates the parameters. Instead, all coefficients (including the intercept) are fitted simultaneously. Using Ordinary Least Squares (OLS), we find coefficient estimates that minimize the sum of the squared errors in the dependent variable. That is, we minimize the vertical distance between the model's predicted Y value at a given location in X and the observed Y value there. To find a vector of beta estimates, we use the following matrix equation:
$$ \boldsymbol{\hat\beta} = \bf (X^\top X)^{-1}X^\top Y $$ It is worth noting explicitly that the coefficients we find this way will not necessarily be the same as those betas found individually. To understand this further, it may help you to read my answer here: Is there a difference between 'controlling for' and 'ignoring' other variables in multiple regression?

At any rate, the standard errors for a multiple regression model are calculated as:
$$ SE_\boldsymbol{\hat\beta} = \sqrt{{\rm diag}\{ s^2\bf (X^\top X)^{-1}\}} $$ where $s^2$ is the variance of the residuals and $\rm diag$ refers to extracting the elements on the main diagonal of the matrix. Since the intercept ($\hat\beta_0$) is first of our regression parameters, it is the square root of the element in the first row first column.

Once we have our fitted model, the standard error for the intercept means the same thing as any other standard error: It is our estimate of the standard deviation of the sampling distribution of the intercept. For a fuller description of standard errors in a regression context, it may help to read my answer here: How to interpret coefficient standard errors in linear regression?

A common use of the intercept's standard error would be to test if the observed intercept is reasonably likely to have occurred under the assumption that its true value is some pre-specified number (such as $0$), as @IrishStat notes.

$\endgroup$
6
  • $\begingroup$ You said "That is, we minimize the vertical distance between the model's predicted Y value at a given location in X and the observed Y value there" . Did you mean "That is, we minimize the sum of the squares of the vertical distances between the model's predicted Y value at a given location in X and the observed Y value there based upon all observations." as compared to least absolute deviation (LAD) $\endgroup$
    – IrishStat
    Commented Sep 19, 2015 at 23:05
  • $\begingroup$ @IrishStat, I was trying to rephrase the initial statement in a more intuitive, but less mathematically rigorous way. I meant squared distances, not absolute distances. $\endgroup$ Commented Sep 19, 2015 at 23:11
  • $\begingroup$ @gung-ReinstateMonica Do you have any references for how you define the standard error? It's so clean and easy to compute, but I haven't found any books / papers that use this formulation. In addition, the results using your method differ slightly (to maybe 2-3 decimal places) compared to standard statistical packages. $\endgroup$
    – PyRsquared
    Commented Nov 26, 2021 at 11:21
  • $\begingroup$ @PyRsquared, just any regression textbook. I probably copied this out of Kutner et al. Applied Linear Statistical Models (which I don't have with me right now). You can also see it on the Wikipedia page (although the notation is a little different). Statistical packages should give you the same values, but they often use slightly different computations to enhance numerical stability. $\endgroup$ Commented Nov 26, 2021 at 12:42
  • $\begingroup$ @gung-ReinstateMonica Thanks for the quick reply. Ok that's great - I also saw the same formulation here where the post mentions the book "Applied Linear Regression by Draper and Smith" $\endgroup$
    – PyRsquared
    Commented Nov 26, 2021 at 12:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.