# How to estimate the variance of the regression coefficients via Variance-Covariance Matrix?

everyone

I am not used to the matrix algebra. I know the maximum likelihood method to get the point estimation of the regression coefficients well. However, I don't understand the mathematical method behind to get the interval estimation of the regression coefficient.

For example, if we construct a logistic regression model, how do we get the variance of the beta regression coefficients?

I searched the Internet and there are some discussions which mention the Variance-Covariance Matrix method. But what is the connection between the matrix and the variance of the regression coefficients?

My question is: How do we use the Variance-Covariance Matrix to derive the variance of the regression coefficients of a logistic regression model? Can you briefly post the steps and rationales of this method? Thank you very much.

You help is much appreciated.

PS: I am not a statistician.

Tom

• See this answer where it says: “ This matrix holds the variances in the diagonal elements and covariances in the off-diagonal elements.” It’s in the context of linear regression, but it’s true for other models.
– EdM
Commented Jul 24 at 14:55
• Thank you for the info. Although it is difficult for me to understand the whole steps, I have a preliminary understanding of the rationale, after spending an hour on the post. I will update after several days about my progress. Much appreciated! Commented Jul 25 at 13:14
• @Edm How does this method apply for solving the same problem of a logistic regression? There is no error term in a logistic regression model. Commented Jul 26 at 9:16
• The ordinary least squares formula is a specific case of maximum-likelihood estimates used in generalized linear models like logistic regression. See this answer, for example. In general, the variance-covariance matrix (vcov) is the inverse of the matrix of second derivatives of the likelihood with respect to the parameters. In logistic regression the likelihood function itself includes the assumed binomial variance (error) of observations. Coefficient standard errors are still the square roots of the diagonal elements of vcov.
– EdM
Commented Jul 26 at 13:30
• Thanks. It's a little difficult for me to understand. Now I know how the expression of b comes in the form of the matrix. The next step is to understand why b has a normal distribution. Commented Jul 28 at 5:37

Estimating a single parameter value

Even when estimating only a single parameter via maximum likelihood, the normal distribution of the parameter estimates around the true value is an asymptotic result, holding in the limit of an infinite number of observations. This answer shows how that asymptotic result can be derived from the central limit theorem or the mean value theorem. The normal distribution, if it holds, simplifies setting of confidence intervals and significance tests as they come directly from the properties of normal distributions.

There is no assurance that the normal distribution of parameter estimates holds with small sample sizes, however. Evaluating the profile of the likelihood as a function of the parameter value can be more reliable, although it requires re-fitting the model over a range of assumed parameter values. See this page for a summary of different tests for models fit by maximum likelihood.

This page shows, for the single parameter case, that the variance of the asymptotically normal distribution is related to the inverse of the Fisher information at the true value of the parameter. The Fisher information is related to the second derivative of the log-likelihood with respect to the parameter value. In generalized linear models the form of the log-likelihood depends on the assumed family and link function. Quoting from Wikipedia:

Thus, the Fisher information may be seen as the curvature of the support curve (the graph of the log-likelihood). Near the maximum likelihood estimate, low Fisher information therefore indicates that the maximum appears "blunt", that is, the maximum is shallow and there are many nearby values with a similar log-likelihood. Conversely, high Fisher information indicates that the maximum is sharp.

In practice, there's yet another approximation involved even in the single-parameter case. You don't know the true value of the parameter, so you approximate the true (asymptotic) normal distribution as centered at your point estimate, with the inverse of the observed Fisher information (at that point estimate) determining the variance.

Multiple parameters

This answer illustrates the extension of the single-parameter to the multiple-parameter case. The Fisher information is now related to the matrix of the second derivatives of the log-likelihood with respect to pairwise combinations of predictors. You use the vector of point estimates and the matrix inverse of the observed Fisher information matrix to estimate the joint (asymptotic) multivariate normal distribution of the parameter estimates.

In multiple linear regression, the Fisher information has the simple form $$X^TX/\sigma^2$$, where $$X$$ is the design matrix and the true variance $$\sigma^2$$ is assumed constant for all observations. Its inverse gives the well-known formula for the multiple linear regression variance-covariance matrix, $$\sigma^2(X^TX)^{-1}$$. For binary data, McCullagh and Nelder show in Section 4.4 that the Fisher information matrix has the form $$X^TWX$$, where $$W$$ is a diagonal weight matrix related to the true probabilities $$\pi_i$$ for the $$m_i$$ observations corresponding to each row $$i$$ of the design matrix (and thus to the expected variance among observations). Unlike for multiple linear regression, there's no general way to simplify the variance-covariance matrix $$(X^TWX)^{-1}$$ to separate out the contribution of the design matrix $$X$$ itself.