Some statistical software use the Wald statistic when reporting on the regression coefficients. As examples, R and Stata report Wald by default.

The logistic regression article on Wikipedia says, unfortunately without reference:

“Rather than the Wald method, the recommended method to calculate the p-value for logistic regression is the likelihood-ratio test (LRT)”

How are Wald and LRT calculated for a logistic regression coefficient (independent variable)? This is for a reminder on how they are calculated and to highlight their differences.

From the Wald test page on Wikipedia:

  • [T]he Wald test is not invariant to a reparametrization, while the Likelihood ratio tests will give exactly the same answer whether we work with R, log R or any other monotonic transformation of R.

So in the context of logistic regression, if you logged a regressor its p-value would be different compared to if it was unlogged (is this correct). Would the p-value change if the LRT was used?

From the same Wald test page:

  • The other reason is that the Wald test uses two approximations (that we know the standard error, and that the distribution is χ2), whereas the likelihood ratio test uses one approximation (that the distribution is χ2).

Although Wald and likelihood ratio are asymptotically equivalent, in the logistic regression we are usually in the pre-asymptote setting, so this is not a reason to view then as equivalent.

Thus it seems that the Wald test disadvantages outweigh the advantages in the logistic setting, and the likelihood ratio is better.

It is my guess that the Wald test is used by logistic regression software routines for its easier computational efficiency, which was more important in the past when software such as R and Stata were first created. Then, through backwards compatibility and not wanting to change the semantics of their logistic functions, the Wald statistic has remained the default. Is there any evidence for this being the case?

Should I be changing the default Wald to likelihood ratio? A lesser question, is there an easy was to do this in R?

  • 1
    $\begingroup$ You have the covariance matrix for the coefficient estimates to hand after fitting a model, so performing Wald tests on $p$ coefficients is just a matter of taking $p$ square roots. Performing the corresponding likelihood-ratio tests, on the other hand, would require fitting $p$ new models. Given that the analyst mightn't be interested in performing any tests at all, it's probably for the best that software displays the results Wald tests in default model aummaries. $\endgroup$ Mar 7 at 0:49
  • $\begingroup$ @Scortchi-ReinstateMonica this reasons is an excellent one for why Wald tests are the default. It could be the core part of an answer. I know that the Wald test and logged LRT measure respectively the horizontal and vertical distance between the maximum likelihood and null hypothesis value, but would like a clarification and context of the main differences they give you in the logistic regression setting, one of which would be how the Wald p-value is already nearly pre-calculated from the covariance matrix. $\endgroup$ Mar 7 at 8:16

In logistic regression (& other generalized linear models with canonical link functions), the coefficient estimates $\hat\theta$ are arrived at by Fisher Scoring: iterating $$\vec\theta_{k+1} = \vec\theta_k + \mathcal{I}^{-1}(\vec\theta_k)U(\vec\theta_k)$$ where $\mathcal{I}$ is the Fisher information & $U$ the score, until convergence. When you're done, you're left with the covariance matrix $\mathcal{I}^{-1}$ for the coefficient estimates; the square roots of its diagonal elements are the variances you need for Wald tests of each coefficient. So you get Wald tests for free, almost, just by fitting a model; but likelihood-ratio tests require fitting a new model for each coefficient you want to test—with a large sample size & many predictors they'd take a good while longer to conduct. (This is also true more generally: if you're using observed information (the negative Hessian of the log likelihood) rather than expected information; or even if you're finding maximum likelihood estimates with an algorithm that doesn't involve calculating the Hessian, it's quicker to evaluate the Hessian numerically than to fit lots of models.)

If the point of logistic regression were to always test whether each & every coefficient is equal to zero, then there'd be an argument for statistical software's defaulting to the likelihood-ratio test when displaying a summary of the fitted model. But as that isn't always, or even often, the point—& especially as with some models many of the hypotheses tested may well be of no interest at all in general (see What value does one set, in general, for the null hypothesis of β0 in a linear regression model?)— it makes sense to provide the Wald tests & leave the analyst to choose which, if any, further tests to conduct, & what method to use. (It would also make sense to provide no tests, & force the analyst to think about which, if any, tests to conduct, &c.)

† I don't know of any R function to conduct LRTs for all coefficients of a model individually—it wouldn't be hard to write one—but both stats:::drop1 & car:::Anova conduct them for a default set of null hypotheses more likely to be of interest.

NB invariance to reparametrization means only that the LRT for, say, $H_0: \beta_7 =0$ is the same as the LRT for $H_0: \frac{1}{1+\mathrm{e}^{-\beta_7}}=1$ (which isn't the case for the Wald test). Replacing $\beta_7$ with $\log \beta_7$, on the other hand, would be fitting a substantively different model.

  • 1
    $\begingroup$ For complete understanding could you include where the first equation comes from? I had assumed that MLE would be used to calculate the coefficients, but the above equation is somehow related to this, if the canonical link function is used? $\endgroup$ Mar 8 at 14:21
  • 1
    $\begingroup$ @SingleMalt: It's finding the maximum likelihood estimates by finding the root of the score equation numerically - there isn't an analytic solution. I added a link - if you're familiar with Newton-Raphson you'll get how it works. There's a good illustration here. $\endgroup$ Mar 8 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.