What is the interpretation of the covariance of regression coefficients?

The lm function in R can print out the estimated covariance of regression coefficients. What does this information give us? Can we now interpret the model better or diagnose issues that might be present in the model?

• The same interpretation as all other covariances---linear covariation? The main use is to calculate the variance of selected contrasts of interest, for instance to test contrasts. May 15 '15 at 21:48

The most basic use of the covariance matrix is to obtain the standard errors of regression estimates. If the researcher is only interested in the standard errors of the individual regression parameters themselves, they can just take the square root of the diagonal to get the individual standard errors.

However, often times you may be interested in a linear combination of regression parameters. For example, if you have a indicator variable for a given group, you may be interested in the group mean, which would be

$\beta_0 + \beta_{\rm grp}$.

Then, to find the standard error for that group's estimated mean, you would have

$\sqrt{X^\top S X}$,

where $X$ is a vector of your contrasts and $S$ is the covariance matrix. In our case, if we only have the addition covariate "grp", then $X = (1,1)$ ($1$ for the intercept, $1$ for belonging to the group).

Furthermore, the covariance matrix (or more over, the correlation matrix, which is uniquely identified from the covariance matrix but not vice versa) can be very useful for certain model diagnostics. If two variables are highly correlated, one way to think about it is that the model is having trouble figuring out which variable is responsible for an effect (because they are so closely related). This can be helpful for a whole variety of cases, such as choosing subsets of covariates to use in a predictive model; if two variables are highly correlated, you may only want to use one of the two in your predictive model.

• Thank you for the explanation. In your last paragraph you are describing the issues that can arise when independent variables are highly collinear. It seems like it would be easier to look at the covariance/correlation of actual $X$s than $\beta$s. $$Var(\hat\beta)=E(\hat\varepsilon^2)\left(X^\prime X\right)^{-1}$$ there is an inverse in the formula.
– mss
May 18 '15 at 16:27

There are two "kinds" of regression coefficients:

1. "True" regression coefficients (usually denoted $\beta$) that describe the underlying data-generating process of the data. These are fixed numbers, or "parameters." An example would be the speed of light $c$, which (we assume) is always the same everywhere in the accessible universe.
2. Estimated regression coefficients (usually denoted denoted $b$ or $\hat \beta$) that are calculated from samples of the data. Samples are collections of random variables, so estimated regression coefficients are also random variables. An example would be an estimate for $c$ obtained in an experiment.

Now think about what covariance means. Take any two random variables $X$ and $Y$. If $\left| \mathrm{Cov}\left(X,Y\right) \right|$ is high, then whenever you draw a large absolute value of $X$ you can also expect to draw a large absolute value of $Y$ in the same direction. Note that "high" here is relative to the amount of variation in $X$ and $Y$, as pointed out in the comments.

The (estimated) covariance of two regression coefficients is the covariance of the estimates, $b$. If the covariance between estimated coefficients $b_1$ and $b_2$ is high, then in any sample where $b_1$ is high, you can also expect $b_2$ to be high. In a more Bayesian sense, $b_1$ contains information about $b_2$.

Note again that "high" is relative. Here "$b_1$ is high" means that "$b_1$ is high relative to its standard error," and their covariance being "high" mean "high relative to the product of their standard errors." One way to smooth out these interpretive hiccups is to standardize each regression input to by dividing by its standard deviation (or two standard deviations in some cases).

One user on this site described $\mathrm{Cov}\left(b_1,b_2\right)$ as "a bit of a fudge," but I don't entirely agree. For one thing, you could use this interpretation to come up with informative priors in Bayesian regression.

As for what this is actually used for, Cliff AB's answer is a good summary.

• This is nice, but I'm a little bothered about the interpretation of the covariance as if it were a correlation. I know you know the difference, but it doesn't come across clearly. I'm also glad you have challenged the "bit of a fudge" comment, because that was a misleading assessment (in an otherwise fine answer). Indeed, the covariance of $b_i$ and $b_j$ for $i\ne j$ gives fundamental and useful information about how those estimates are interrelated, as @Cliff AB indicates.
– whuber
May 16 '15 at 20:14
• @whuber thanks, and I did actually write "correlation" at one point. I'll clean it up when I get off my phone May 16 '15 at 20:20
• Since I might not make it back to this thread for a while, +1 in advance for the edits!
– whuber
May 16 '15 at 20:24
• made the same mistake in my description! May 16 '15 at 21:57
• @whuber now I'm actually second guessing my own understanding of covariance. Is my issue just that I didn't emphasize the fact that the scales could be different, or am I missing something else? I came across your "boxes" explanation and I don't see what that might be May 16 '15 at 22:12