5
$\begingroup$

I have a bunch of data that I fit a linear regression to, and now I need to find the variance of my slope. Is there an analytical way to get this?

If an example is necessary, consider this my data in R:

x <- c(1:6)
y <- c(18, 14, 15, 12, 7, 6)
lm(y ~ x)$coefficients

So I have a slope estimate of -2.4, but I want to know the variance of that estimate.

After looking at previous questions, I've seen a few equations for estimating the slope parameter, but I'm a little confused about what the differences between equations are and what approach is valid for my problem.

For example, the answers in this question say that $\newcommand{\Var}{\rm Var}\newcommand{\slope}{\rm slope}\Var[\slope] = \frac{V[Y]}{\sum\left(\frac{x_i-\bar{x}}{\sum(x_i-\bar{x})^2}\right)}$.

This question says that $\Var[\slope] = \frac{V[Y]}{\sum(x_i-\bar{x})^2}$.

And if I look at the output in R (as a "check" mechanism), I'm given two other ways I could potentially calculate the slope variance (one using the standard error, another given the covariance matrix). I feel like I'm missing something key because all these estimates give me similar (but not the same) answer.

$\endgroup$
3
  • 1
    $\begingroup$ Are you wondering how to get R to give this to you? (You would use ?vcov(model).) Or are you wondering where that comes from / why? $\endgroup$ Mar 30, 2015 at 15:50
  • $\begingroup$ I'm less interested in R tricks, and more interested in how I would calculate the variance myself (but using R as a calculator of sorts). Does that make sense? I want to know how to find the variance of a slope, what formula makes sense, and not so much how can I find the answer using R. $\endgroup$
    – user72320
    Mar 30, 2015 at 15:54
  • 1
    $\begingroup$ I think the answer you want may be at the linked thread. Please read it. If it isn't what you're want / you still have a question afterwords, come back here & edit your Q to state what you learned & what you still need to know. Then we can provide the information you need w/o just duplicating material elsewhere that already didn't help you. $\endgroup$ Mar 30, 2015 at 16:28

2 Answers 2

5
$\begingroup$

For a standard linear regression that meets the normal assumptions, the variances of your parameter estimates can be taken from the variance covariance matrix, $\Sigma$. For example, the variance of the intercept is the first element on the main diagonal, $\Sigma_{11}$. The variance of the slope on $X_1$ is the second element on the main diagonal, $\Sigma_{22}$, and so on.

There are probably many ways to skin a cat, but the standard calculation for the variance covariance matrix uses the residual variance from your model and your design matrix. Then it is:
$$ \rm{VCov(model)} = s^2(X' X)^{-1} $$ Here's a worked example of the calculations with your data and R:

x = c(1:6);    y = c(18, 14, 15, 12, 7, 6);    m = lm(y ~ x)
summary(m)
# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept)  20.4000     1.4119   14.45 0.000133 ***
# x            -2.4000     0.3625   -6.62 0.002700 ** 
# 
# Residual standard error: 1.517 on 4 degrees of freedom
s = summary(m)$sigma;  s  # [1] 1.516575
dm = model.matrix(m);  dm
#   (Intercept) x
# 1           1 1
# 2           1 2
# 3           1 3
# 4           1 4
# 5           1 5
# 6           1 6
s^2*solve(t(dm)%*%dm)
#             (Intercept)          x
# (Intercept)    1.993333 -0.4600000
# x             -0.460000  0.1314286
vcov(m)  # you can see that this is the same as the manual calculation above
#             (Intercept)          x
# (Intercept)    1.993333 -0.4600000
# x             -0.460000  0.1314286
sqrt(diag(vcov(m)))  # these are the same standard errors as the summary output
# (Intercept)           x 
#   1.4118546   0.3625308 
$\endgroup$
2
  • $\begingroup$ I follow that, and I understand and agree that that's how you would manually get the variance/covariance matrix, but I still don't understand the equations in those other questions. The method in your answer gives one result, but those two equations give a different one. Is there some scaling factor the variance covariance matrix doesn't take into account? $\endgroup$
    – user72320
    Mar 31, 2015 at 14:14
  • $\begingroup$ @user72320, you can see that both the variance covariance matrix & the manual calculation give exactly the right answer. It the equations from the other Qs differ, they are wrong. Note also that Glen_b's answer below gives the proof for the formula used here. $\endgroup$ Mar 31, 2015 at 14:56
4
$\begingroup$

It's easy to do this for the multiple regression case: \begin{align} \text{Var}((X'X)^{-1}X'y) &= (X'X)^{-1}X'\text{Var}(y)X'(X'X)^{-1} \\ &=(X'X)^{-1}X'(\sigma^2I)X'(X'X)^{-1} \\ &=\sigma^2(X'X)^{-1}X'X'(X'X)^{-1} \\ &=\sigma^2(X'X)^{-1} \end{align} which is usually estimated by $s^2(X'X)^{-1}$

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.