Variance between two coefficients in linear regression model

Question

I am trying to figure out the variance of the difference between two coefficients in a linear regression model. If I am given the design matrix (X^T X) and the value of sigma, how do I go about solving: var(beta.1.hat - beta.2.hat)? My thought process is finding the variance for each part using the formula var(beta.j.hat) = sigma^2((X^T X)^-1 subscript jj. Then var(beta.1.hat - beta.2.hat) should be equal to: var(beta.1.hat) + var(beta.2.hat) - 2 Cov(beta.1.hat,beta.2.hat)

Is my logic correct? Please give me some suggestions or assistance thanks!

Answer 1

Assuming you have 2 variables, write $\beta = (\beta_1, \beta_2)^T$ and let $c = (1, -1)^T$. Recall that in ordinary least squares, $Var(\hat\beta) = \sigma^2 (X^TX)^{-1}$. \begin{align*} Var(\hat \beta_1 - \hat \beta_2) &= Var(c^T\hat\beta)\\ &= c^TVar(\hat\beta)c\\ &=\sigma^2 c^T(X^TX)^{-1}c \end{align*} This reduces to what you have, so your logic is correct.

If you don't remember the variance (or other parts of least squars. Posit model $Y=X\beta + \epsilon$ where $\epsilon \sim (0, \sigma^2I)$. Minimize the following objective function: \begin{align*} f(\beta) &= (Y-X\beta)^T(Y-X\beta)\\ f'(\beta) &= -2X^T(Y-X\beta) = 0\\ \hat\beta &= (X^TX)^{-1}X^TY\\ Var(\hat\beta) &= Var((X^TX)^{-1}X^TY)\\ &= (X^TX)^{-1}X^T Var(Y) X(X^TX)^{-1}\\ &= (X^TX)^{-1}X^T Var(X\beta + \epsilon) X(X^TX)^{-1}\\ &= (X^TX)^{-1}X^T Var(\epsilon) X(X^TX)^{-1}\\ &= (X^TX)^{-1}X^T \cdot \sigma^2I \cdot X(X^TX)^{-1}\\ &= \sigma^2(X^TX)^{-1} \end{align*} Note: normality assumption is not required unless you need to estimate $\sigma^2$ and/or do inference.