4
$\begingroup$

Consider the following two logistic regression models: $$ \begin{aligned} &\text{Model A: }&P(Y=1)&=\frac{\text{exp}\left(b_1+b_2X_2\right)}{1+\text{exp}\left(b_1+b_2X_2\right)} \\ &\text{Model B: }&P(Y=1)&=\frac{\text{exp}\left(b_1+b_2X_2+b_3X_3\right)}{1+\text{exp}\left(b_1+b_2X_2+b_3X_3\right)} \end{aligned} $$ In model A, suppose that both the parameters $b_1$ and $b_2$ appear significant, i.e. $-1,96 > z_k$ or $1,96 < z_k$. When adding a dummy variable $X_3$ to create model B, there seem to be two options if I want to check whether the variable adds more explanatory power to the model:

  1. Perform a Likelihood-ratio test between model A and model B by calculating the test statistic $G^2=-2\log \left(\frac{L_{\text{A}}}{L_{\text{B}}}\right)$ and compare it to the critical value of the chi-squared distribution with one degree of freedom.
  2. Perform a z-test of $X_3$ in model B, calculating $z=\frac{b_3}{\text{se}\left(b_3\right)}$ and compare it to the critical values of the standard normal distribution.

The question now is: Is it sufficient in this case to calculate the z-value, if I want to determine whether model B has significant more explanatory power than model A? Is it possible that $X_3$ in a z-test is significant, and at the same time the LR-test implies there is no difference between the two models?

In summary: when adding only one variable to a logistic model, what is the best test to perform, if I want to investigate if the variable added significant explanatory power to the model?

$\endgroup$
3
$\begingroup$

Under the null hypothesis $b_3=0$, the Wald z-test assumes Normality of the coefficient estimate $$\DeclareMathOperator{\se}{se} \frac{\hat b_3}{\se\left(\hat b_3\right)}\sim\mathcal{N}(0,1)$$ while Wilk's likelihood ratio test assumes merely that there exists some transformation to Normality$g(\cdot)$ $$\frac{g\left(\hat b_3\right)}{\se\left(g\left(\hat b_3\right)\right)}\sim\mathcal{N}(0,1)$$ Pawitan (2001), In all Likelihood $\S$ 2.9

If you plot the log-likelihood of $b_3$ (in the case you described with the Wald test significant & the LRT not) you'll probably find it's not much like a parabola, & therefore Wald's test would be likely to over-estimate significance compared to Wilk's.

As @Stask says, the two are equivalent asymptotically; it's just that the LRT, by acting as if it were choosing the best Normalizing transformation, approaches Normality quicker.

$\endgroup$
  • $\begingroup$ Great and clear explanation, thanks! I think it comes down to the understanding of what exactly is going on when performing an LR test, i.e. what the assumption "transformation to normality" means in mathematical terms. Your conclusion about the Wald's test overestimating is making the process a lot more transparent. $\endgroup$ – thesixmax Jan 28 '13 at 14:38
1
$\begingroup$

The two tests are asymptotically equivalent, i.e., in large samples, should produce similar answers. See the classic expository note by Buse (1982). The Wald test ($z$-test) relies on large samples just as much as the likelihood ratio does, so one can't say that one is better justified than the other. However, my understanding is that in small samples, likelihood ratio may perform a little better than the Wald test (can't really back this up with references, this is just the word of mouth that I tend to hear). On the other hand, likelihood ratio testing crucially hinges on i.i.d. assumptions, while with Wald test, all you need is a consistent standard error, so Wald tests are applicable in a broader set of situations, including say correlated data and GEE.

$\endgroup$
  • $\begingroup$ Yes I agree. Some days ago I read a book on statistical methods that stated: "A Wald test should never stand alone". I did not know that the Wald test did not rely upon i.i.d assumptions - how could that be the case? Should the observations not be i.i.d to satisfy the CLT, when performing an z-test for example? $\endgroup$ – thesixmax Jan 28 '13 at 14:47
  • 1
    $\begingroup$ For the strict form of the CLT, you do need i.i.d. data. However, there are weaker forms, i.e., you can get asymptotic normality under weaker conditions. If you want a formal treatment, see work on robust statistics by Huber, starting from his classic 1967 paper; or more generally on M-estimation. $\endgroup$ – StasK Jan 28 '13 at 16:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.