1
$\begingroup$

I am trying to figure out the variance of the difference between two coefficients in a linear regression model. If I am given the design matrix (X^T X) and the value of sigma, how do I go about solving: var(beta.1.hat - beta.2.hat)? My thought process is finding the variance for each part using the formula var(beta.j.hat) = sigma^2((X^T X)^-1 subscript jj. Then var(beta.1.hat - beta.2.hat) should be equal to: var(beta.1.hat) + var(beta.2.hat) - 2 Cov(beta.1.hat,beta.2.hat)

Is my logic correct? Please give me some suggestions or assistance thanks!

$\endgroup$
1
  • 1
    $\begingroup$ If you know the variance of $Y$, then you could multiply $\beta$ by a matrix $[1, -1]$ and apply the same to the least squares estimator and calculate the variance as such. $\endgroup$ – user275978 Mar 8 '20 at 0:21
0
$\begingroup$

Assuming you have 2 variables, write $\beta = (\beta_1, \beta_2)^T$ and let $c = (1, -1)^T$. Recall that in ordinary least squares, $Var(\hat\beta) = \sigma^2 (X^TX)^{-1}$. \begin{align*} Var(\hat \beta_1 - \hat \beta_2) &= Var(c^T\hat\beta)\\ &= c^TVar(\hat\beta)c\\ &=\sigma^2 c^T(X^TX)^{-1}c \end{align*} This reduces to what you have, so your logic is correct.

If you don't remember the variance (or other parts of least squars. Posit model $Y=X\beta + \epsilon$ where $\epsilon \sim (0, \sigma^2I)$. Minimize the following objective function: \begin{align*} f(\beta) &= (Y-X\beta)^T(Y-X\beta)\\ f'(\beta) &= -2X^T(Y-X\beta) = 0\\ \hat\beta &= (X^TX)^{-1}X^TY\\ Var(\hat\beta) &= Var((X^TX)^{-1}X^TY)\\ &= (X^TX)^{-1}X^T Var(Y) X(X^TX)^{-1}\\ &= (X^TX)^{-1}X^T Var(X\beta + \epsilon) X(X^TX)^{-1}\\ &= (X^TX)^{-1}X^T Var(\epsilon) X(X^TX)^{-1}\\ &= (X^TX)^{-1}X^T \cdot \sigma^2I \cdot X(X^TX)^{-1}\\ &= \sigma^2(X^TX)^{-1} \end{align*} Note: normality assumption is not required unless you need to estimate $\sigma^2$ and/or do inference.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.