6
$\begingroup$

This question might sound simple, but in fact I am absolutely unsure whether it is allowed to use an LRT for model comparison when the error distributions differ.

For example, can I compare a model with a gaussian distribution with a gamma or a log-gamma model?

$\endgroup$
2
  • 2
    $\begingroup$ I don't believe that would make sense. A likelihood ratio is only meaningful for comparing likelihoods derived from the same model. If you could just pick any two models to compare, you would be able to game the test and get whatever results you desired. You can, however, compare nested models; i.e., where one model is a restriction on another. For example, an exponential model in event history is equivalent to a Weibull with p=1. $\endgroup$ Commented Nov 10, 2012 at 22:22
  • 1
    $\begingroup$ Dear Jason, perfect answer ! AS I wrote in the comment to Glen_b I saw already that comparing nested models is possible. One more question. Should the direction be "complicated" nested in "less complicated" or the other way around? $\endgroup$
    – Jens
    Commented Nov 12, 2012 at 9:17

2 Answers 2

3
$\begingroup$

This is more a set of rambling comments and thoughts than an actual answer, but it's a bit long for a comment. It might end up as something approximating an answer of at least some slight value eventually, though, if it is edited enough times by me or other people.

Generally, the Neyman-Pearson Lemma doesn't apply except in special cases, so the base answer is 'well, no'.

However, I am going to go on a ramble that basically says 'Clearly in general you won't have an asymptotic chi-square distribution, but maybe there's something to looking at the ratio of likelihoods'.

To begin - I think the Likelihood Principle should apply, which might at least give us some hope that the ratio of likelihoods could be informative about the problem.

I've seen it done (use a ratio of likelihoods to derive a statistic) for goodness of fit tests (testing a specific null against a specific alternative) - but just to get a form of statistic rather than its distribution. In a number of particular cases (e.g. some specific symmetric nulls vs specific symmetric alternatives), the likelihood ratio does often seem to lead to a very sensible test statistic, one that has excellent power properties. Alternatively efficient scores have been used as a way to get to test statistics that sometimes end up looking like (a monotonic function of) a likelihood ratio, again, suggesting that likelihood ratios may be informative in testing one distribution against another.

Then again, I've also seen people claim that you can't do that kind of thing at all; that the ratio of likelihoods isn't meaningful.

From a Bayesian point of view such comparisons seem to present no immediately obvious problem (unless I missed something, which certainly happens), as long as everything is appropriately normalized. With equal prior probability, we boil down to looking at the Bayes factor. If instead of the Bayes factor integral, the likelihood corresponding to the maximum likelihood estimate of the parameter for each model is used, then we get back to a likelihood ratio, as mentioned here. Alternatively, we might look at approximating the integrals using Laplace's method -- at least in some circumstances the likelihood ratio can come up as a term in it, though there's another factor there; such things are at least suggestive that the likelihood ratio is the appropriate way to make use of the likelihood principle, even if we don't have a distribution for the ratio.

For Gaussian vs gamma, you can parameterize that (note that they're both special cases of the Tweedie distribution), so that may make a difference even if the general case isn't okay, though in the Tweedie family the Gaussian is a rather special case, since there's a "boundary" there (as there is for the Poisson), so again, it's not a standard situation - indeed, there's a 'gap' in the parameter space between the Gaussian and the gamma.

$\endgroup$
1
  • $\begingroup$ many thanks for your thoughts. I remember that in Zuur et al. 2009 (Mixed Effects modelling book) they were comparing "nested" distributions, i.e. negative binomial and poisson. Yet I think I would rather go for your "well, no" answer. When I compared to models the deviances for model 1 was 4000 and for model 2 was 5000000 (or something like that), thus it did not look very convenient to compare both models. $\endgroup$
    – Jens
    Commented Nov 12, 2012 at 9:14
2
$\begingroup$

I think that you understand that nested comparisons are well understood using the LRT.

Non-nested comparisons are also possible. Consider two alternative distributions, one parameterized by $\theta$, $f_\theta(x)$, the other parameterized by $\eta$, $f_\eta(x)$, with $\theta$ and $\eta$ of equal length (this condition can be dropped; see the link below).

Take the differences in the log likelihoods: $$\begin{equation*} T(X) = \frac{1}{n}[l(\theta) - l(\eta)] = \frac{1}{n} \sum_{i=1}^n{\log\left(\frac{f_\theta(x_i)}{f_\eta(x_i)}\right)}. \end{equation*}$$ By the law of large numbers, this mean converges to it expected value. If the two distributions model the process equally well, the expected value is 0. By the central limit theorem, the distribution is normal. The only difficulty lies in finding the variance.

I would estimate the variance by bootstrapping values of the test statistic $T(X)$. Actually, I'd just bootstrap the distribution of $T(X)$.

This is known as the Vuong test. It can be extended to the partially nested case as well.

As a broader comment, you can use any test statistic that you want to test a hypothesis. The only questions are:

  • What is the distribution of the test statistic under the null?
  • How powerful is this test statistic relative to alternatives that you care about?
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.