How to estimate the variance of the regression coefficients via Variance-Covariance Matrix?

Question

everyone

I am not used to the matrix algebra. I know the maximum likelihood method to get the point estimation of the regression coefficients well. However, I don't understand the mathematical method behind to get the interval estimation of the regression coefficient.

For example, if we construct a logistic regression model, how do we get the variance of the beta regression coefficients?

I searched the Internet and there are some discussions which mention the Variance-Covariance Matrix method. But what is the connection between the matrix and the variance of the regression coefficients?

My question is: How do we use the Variance-Covariance Matrix to derive the variance of the regression coefficients of a logistic regression model? Can you briefly post the steps and rationales of this method? Thank you very much.

You help is much appreciated.

PS: I am not a statistician.

Tom

Answer 1

Estimating a single parameter value

Even when estimating only a single parameter via maximum likelihood, the normal distribution of the parameter estimates around the true value is an asymptotic result, holding in the limit of an infinite number of observations. This answer shows how that asymptotic result can be derived from the central limit theorem or the mean value theorem. The normal distribution, if it holds, simplifies setting of confidence intervals and significance tests as they come directly from the properties of normal distributions.

There is no assurance that the normal distribution of parameter estimates holds with small sample sizes, however. Evaluating the profile of the likelihood as a function of the parameter value can be more reliable, although it requires re-fitting the model over a range of assumed parameter values. See this page for a summary of different tests for models fit by maximum likelihood.

This page shows, for the single parameter case, that the variance of the asymptotically normal distribution is related to the inverse of the Fisher information at the true value of the parameter. The Fisher information is related to the second derivative of the log-likelihood with respect to the parameter value. In generalized linear models the form of the log-likelihood depends on the assumed family and link function. Quoting from Wikipedia:

Thus, the Fisher information may be seen as the curvature of the support curve (the graph of the log-likelihood). Near the maximum likelihood estimate, low Fisher information therefore indicates that the maximum appears "blunt", that is, the maximum is shallow and there are many nearby values with a similar log-likelihood. Conversely, high Fisher information indicates that the maximum is sharp.

In practice, there's yet another approximation involved even in the single-parameter case. You don't know the true value of the parameter, so you approximate the true (asymptotic) normal distribution as centered at your point estimate, with the inverse of the observed Fisher information (at that point estimate) determining the variance.

Multiple parameters

This answer illustrates the extension of the single-parameter to the multiple-parameter case. The Fisher information is now related to the matrix of the second derivatives of the log-likelihood with respect to pairwise combinations of predictors. You use the vector of point estimates and the matrix inverse of the observed Fisher information matrix to estimate the joint (asymptotic) multivariate normal distribution of the parameter estimates.

The answer to your question:

... what is the connection between the [variance-covariance] matrix and the variance of the regression coefficients?

is thus as follows. Similarly to how the inverse of the second derivative of the log-likelihood with respect to the parameter leads to the variance of its estimate in the single-parameter situation, in multiple regression the matrix inverse of the observed Fisher information matrix gives the variance-covariance matrix. As for any multivariate normal distribution, the diagonal elements of the variance-covariance matrix are the variances of the corresponding coefficient estimates. The off-diagonal elements are the covariances between the estimates.

Form of the variance-covariance matrix

In multiple linear regression, the Fisher information has the simple form $X^TX/\sigma^2$, where $X$ is the design matrix and the true variance $\sigma^2$ is assumed constant for all observations. Its inverse gives the well-known formula for the multiple linear regression variance-covariance matrix, $\sigma^2(X^TX)^{-1}$. For binary data, McCullagh and Nelder show in Section 4.4 that the Fisher information matrix has the form $X^TWX$, where $W$ is a diagonal weight matrix related to the true probabilities $\pi_i$ for the $m_i$ observations corresponding to each row $i$ of the design matrix (and thus to the expected variance among observations). Unlike for multiple linear regression, there's no general way to simplify the variance-covariance matrix $(X^TWX)^{-1}$ to separate out the contribution of the design matrix $X$ itself.

Using the variance-covariance matrix

The multivariate normal distribution of parameter estimates in general has non-zero off-diagonal elements. To go beyond individual coefficient variances you need to take into account the covariances. That's necessary, for example, in a Wald test on multiple coefficients. It's also necessary when making predictions from the model based on new covariate values, as you need to apply the formula for the variance of a weighted sum of correlated variables to estimate variances for the predictions.