So I'm trying to build my own Wald test and likelihood ratio test code within a machine learning pipeline. I can get the final fitted logistic regression coefficients from liblinear. I'm coding in MATLAB.

How would I get the variance of the regression coefficients and also the confidence intervals of the regression coefficients? Clearly for a variance and confidence interval, you need a sample of multiple sets of coefficients. But I thought you only get a single set of coefficients at the end of the log likelihood optimization.

Basically trying to replicate the results in the following link http://www.ats.ucla.edu/stat/mult_pkg/faq/general/nested_tests.htm

  • 1
    $\begingroup$ You can estimate the variance-covariance matrix for the coefficient estimates from the Hessian of the log-likelihood; you don't need "multiple sets" of coefficients. $\endgroup$ – Glen_b Jun 5 '14 at 23:26
  • $\begingroup$ So for the Wald test, \hat\theta_i is the MLE estimate of my logistic regression coeff \theta_i (which would be the values of \theta_i that I am using for my LR model found through some descent algorithm). \theta_0 = 0, since I'm comparing against the null hypothesis that this coefficient has no effect. var(\hat\theta) is the diagonal of the Hessian matrix of the LR log-likelihood function. Is that right? $\endgroup$ – user2422566 Jun 6 '14 at 20:17
  • 2
    $\begingroup$ Where the Hessian is available, the estimate of the covariance matrix is usually obtained as the inverse of the Hessian. [With Fisher scoring, I think there's an alternative approach that can be used.] $\endgroup$ – Glen_b Jun 7 '14 at 2:38
  • $\begingroup$ Thanks! Also I found some good links here -- public.iastate.edu/~mervyn/stat580/Notes/s09mle.pdf data.princeton.edu/wws509/notes/a1.pdf $\endgroup$ – user2422566 Jun 9 '14 at 18:22

When you fit a logistic regression model, there is no closed form solution for the parameter estimates unlike in linear regression. So instead, you search over the parameter space for a set of parameter estimates that maximize the log likelihood or minimize the deviance. (I usually prefer to think in terms of the deviance, but in this case it might be better to think of maximizing the log likelihood.) The most common search procedure is the Newton-Raphson algorithm.

As you search, you could map out the shape of the log likelihood, but this isn't really done. In the process of running the Newton-Raphson algorithm, you calculate the Hessian matrix (and it shouldn't be difficult to get if you run a search algorithm that doesn't use the Hessian instead). That provides a picture of the shape in the region of the parameter space where you are currently. The Wald test of the parameter is based on the assumption that the log likelihood has the shape of a normal distribution (which it would with infinite data but may not with small samples). An estimate of the standard deviation is calculated, and that is used as the standard error. This is typically what is used to form confidence intervals for betas.

The likelihood ratio test works differently. The ratio of the likelihoods is the difference of the log likelihoods. That is, it is the difference between the likelihoods of the model when a parameter is set at two different values. Essentially always, these two values are the maximum likelihood estimate and the null value (0). The difference between these two values should be distributed as chi-squared. It is less common to use this approach to determine the confidence interval for a beta, but it can be done. You need to search over possible beta estimates and work backward to find values that constitute the limits of the interval.

There is a very useful figure (actually taken from John Fox) at the bottom of the linked page that is extremely helpful for understanding this topic.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.