16
$\begingroup$

I understand that if I have two models A and B and A is nested in B then, given some data, I can fit the parameters of A and B using MLE and apply the generalized log likelihood ratio test. In particular, the distribution of the test should be $\chi^2$ with $n$ degrees of freedom where $n$ is the difference in the number of parameters that $A$ and $B$ have.

However, what happens if $A$ and $B$ have the same number of parameters but the models are not nested? That is they are simply different models. Is there any way to apply a likelihood ratio test or can one do something else?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
12
$\begingroup$

The paper Vuong, Q. H. (1989). Likelihood ratio tests for model selection and non-nested hypotheses. Econometrica, 307-333. has the full theoretical treatment and test procedures. It distinguishes between three situations, "Strictly Non-nested Models", "Overlapping Models", "Nested Models", and also examines cases of misspecification. It is therefore no-accident that it finds that for some cases, the test statistic is distributed as a linear combination of chi-squares.

The paper is not light, neither it proposes an "off-the-shelf" testing procedure. But, for once, its (close to) 3,000 citations speak of its merits, being an inspired combination of classical testing framework and the information-theoretic approach.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I currently facing a similar problem (stats.stackexchange.com/questions/546328/…). I tried to consult the ref you suggest but it is not easy and I don't know if my problem can be easily conducted in their "Strictly Non-nested Models". My guess seems you reasonable? Have you some suggestions? $\endgroup$
    – markowitz
    Commented Oct 2, 2021 at 10:09
  • 1
    $\begingroup$ A caveat about using Vuong's test: "In times past, the Vuong test had been used to test whether a zero-inflated negative binomial model or a negative binomial model (without the zero-inflation) was a better fit for the data. However, this test is no longer considered valid. Please see The Misuse of The Vuong Test For Non-Nested Models to Test for Zero-Inflation by Paul Wilson for further information." stats.oarc.ucla.edu/r/dae/zip $\endgroup$
    – DrJerryTAO
    Commented May 30 at 5:46
  • $\begingroup$ @DrJerryTAO Thanks for pointing this out. The reason why it is not appropriate for this (and perhaps other cases), is that not all of the regularity conditions required by the test are satisfied, as Paul Wilson article explains. This is a good reminder to not automatically assume that these "regularity conditions" are there for mathematical show-off. Especially when some component of the model touches some of the boundaries, the results of maximum likelihood theory change profoundly. $\endgroup$
    – Alecos Papadopoulos
    Commented May 30 at 16:18
  • 1
    $\begingroup$ @AlecosPapadopoulos Another related paper is Molenberghs, G., & Verbeke, G. (2007). Likelihood ratio, score, and Wald tests in a constrained parameter space. The American Statistician, 61(1), 22–27. jstor.org/stable/27643833 for hypothesis that variance is zero, although it does not mention Vuong's paper. Do you think Vuong's test works for the cases described in Molenberghs & Verbeke? $\endgroup$
    – DrJerryTAO
    Commented May 31 at 17:43
  • $\begingroup$ @DrJerryTAO Thanks for the reference. I would say that the general answer is yes, because when the true parameter lies at the boundary (or close to it), it appears (I have seen it elsewhere too), that likelihood statistics behave like a mixture of chi-squares... but most likely each statistic would need its own specific expression as regards the mixture and its variance. $\endgroup$
    – Alecos Papadopoulos
    Commented May 31 at 18:05
1
$\begingroup$

The Generalised likelihood ratio test DOES NOT work the way you are saying. See for example the following lecture notes:

http://www.maths.manchester.ac.uk/~peterf/MATH38062/MATH38062%20GLRT.pdf

http://www.maths.qmul.ac.uk/~bb/MS_Lectures_12b.pdf

The GLRT is defined for hypothesis of the type:

$$H_0: \theta\in\Theta_0 \,\,\, vs. \,\,\, H_1: \theta\in\Theta_1,$$

where $\Theta_0\cap\Theta_1=\emptyset$ and $\Theta_0\cup\Theta_1=\Theta$.

For the framework you describe, you can compare the models using other tools such as AIC and BIC. Also Bayes factors, if you are willing to go full Bayesian.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Welcome to CV. Perhaps it would be of interest to you to look up the paper I am mentioning in my own answer to this question. $\endgroup$
    – Alecos Papadopoulos
    Commented Jul 30, 2014 at 21:33
  • $\begingroup$ @AlecosPapadopoulos Thank you for the reference. I took a quick glance and, as expected, the conditions for that sort of GLRT to work are very very (very very) restrictive. So, I would prefer to go for something safer. I know it is highly cited, apologies for the blasphemy. $\endgroup$
    – Waterman
    Commented Jul 30, 2014 at 21:35
  • $\begingroup$ @AlecosPapadopoulos In particular, I find the compactness of the parameter space condition (Assumption A2) extremely unappealing. $\endgroup$
    – Waterman
    Commented Jul 30, 2014 at 21:40
  • $\begingroup$ The very instructive (although probably not real) historical anecdote around Laplace's magnum opus is that Napoleon the Great read it and commented to Laplace "I see you do not mention God anywhere in your book", to which Laplace supposedly replied "I did not need that hypothesis"... meaning that the concept of "sacred" is not needed in science, and so, there can be no blasphemy. $\endgroup$
    – Alecos Papadopoulos
    Commented Jul 30, 2014 at 21:41
  • 2
    $\begingroup$ ...as for your second comment on Assumption A2, I guess it means that the whole maximum likelihood framework does not quite meet the needs of your field, except perhaps when the distributions involved have log-concave densities. $\endgroup$
    – Alecos Papadopoulos
    Commented Jul 30, 2014 at 22:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.