I understand that in multiple regression analysis, for each independent variable, you would graph dependent variable vs independent variable and you would make a line of best fit and calculate the standard error.

However, how does it work for intercept? Can some one give me a concise but clear explanation?

up vote 2 down vote accepted

The standard error of the the intercept allows you to test whether or not the estimated intercept is statistically significant from a specified(hypothesized) value ...normally 0.0 . If you test against 0.0 and fail to reject then you can then re-estimate your model without the intercept term being present.

Your characterization of how multiple regression works is inaccurate. Your version implies fitting a simple linear regression for each variable in turn (and presumably using each of those slopes as the coefficient for that variable in the multiple regression model). This notion leaves you with the problem of how to deal with the fact that the intercepts from each simple regression are quite likely to differ.

However, that approach is not how multiple regression works / estimates the parameters. Instead, all coefficients (including the intercept) are fitted simultaneously. Using Ordinary Least Squares (OLS), we find coefficient estimates that minimize the sum of the squared errors in the dependent variable. That is, we minimize the vertical distance between the model's predicted Y value at a given location in X and the observed Y value there. To find a vector of beta estimates, we use the following matrix equation:
$$ \boldsymbol{\hat\beta} = \bf (X^\top X)^{-1}X^\top Y $$ It is worth noting explicitly that the coefficients we find this way will not necessarily be the same as those betas found individually. To understand this further, it may help you to read my answer here: Is there a difference between 'controlling for' and 'ignoring' other variables in multiple regression?

At any rate, the standard errors for a multiple regression model are calculated as:
$$ SE_\boldsymbol{\hat\beta} = \sqrt{{\rm diag}\{ s^2\bf (X^\top X)^{-1}\}} $$ where $s^2$ is the variance of the residuals and $\rm diag$ refers to extracting the elements on the main diagonal of the matrix. Since the intercept ($\hat\beta_0$) is first of our regression parameters, it is the square root of the element in the first row first column.

Once we have our fitted model, the standard error for the intercept means the same thing as any other standard error: It is our estimate of the standard deviation of the sampling distribution of the intercept. For a fuller description of standard errors in a regression context, it may help to read my answer here: How to interpret coefficient standard errors in linear regression?

A common use of the intercept's standard error would be to test if the observed intercept is reasonably likely to have occurred under the assumption that its true value is some pre-specified number (such as $0$), as @IrishStat notes.

  • You said "That is, we minimize the vertical distance between the model's predicted Y value at a given location in X and the observed Y value there" . Did you mean "That is, we minimize the sum of the squares of the vertical distances between the model's predicted Y value at a given location in X and the observed Y value there based upon all observations." as compared to least absolute deviation (LAD) – IrishStat Sep 19 '15 at 23:05
  • @IrishStat, I was trying to rephrase the initial statement in a more intuitive, but less mathematically rigorous way. I meant squared distances, not absolute distances. – gung Sep 19 '15 at 23:11

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.