1
$\begingroup$

In the paper "Including Transfer-Out Behavior in Retention Models: Using the NSLC Enrollment Search Data"

(http://www.studentclearinghouse.org/colleges/files/ST_UofMD_casestudy.pdf)

the author compares two models of student retention:

  1. A binary logistic regression (with DV coded as either retained/not retained)
  2. A multinomial logistic regression (with DV coded as retained/not retained/transferred out)

The author uses two criteria for model comparison (p.14):

  1. "Predictive ability" measured by the the log likelihood and model chi-square
  2. "Explanatory power" which is a comparison of which IVs were predictive in each model, and the change in probability associated with a change in each IV.

Based on the goal of the study I understand why criteria #2 is useful for comparison. However, I am wondering if it is correct to compare likelihood ratios between these two types of models. Additionally, are there any other tests that could accurately compare logistic vs. multinomial models?

$\endgroup$
2
$\begingroup$

Criteria 1 does not make any sense in this case. That is because these likelihoods are simply not comparable.

In particular, let's simplify this significantly to make this more clear. Let's forget about explanatory variables and pretend that we have 300 observations. In those 300 observations, 100 are retained, 100 are not retained and 100 transfer out.

In the binary model, we would say 100 are retained and 200 are not retained. This would give us a log-likelihood of

$100 \times \log(1/3) + 200 \times \log(2/3) + \log \left( \frac {300!}{200!100!} \right) = -3.02$

(where did we get these numbers? We have 100 retained subjects who's probability of being in that category is 1/3, and 200 not-retained subjects who's probability of being in that category is 2/3. The final term is the normalizing constant for the binomial distribution: thanks to whuber for pointing that out).

In the multinomial model, we would have log-likelihood

$100 \times \log(1/3) + 100 \times \log(1/3) + 100 \times \log(1/3) + \log \left( \frac{300!}{100!200!} \right) + \log \left( \frac {200!}{100!100!} \right)= -5.90$

So if we look at the log-likelihood to compare models, we see that the binary model is "favored"...even though it clearly contains less information than the multinomial model! If you want an even more extreme example, what if we had (very boringly) used only one group? Then we would have a log-likelihood of 0!

$\endgroup$
  • $\begingroup$ That makes sense. The author uses the likelihood ratio index: 1-(log likelihood of full model/ log likelihood of model only with a constant) as the basis for comparison. If I am understanding correctly, since the log likelihoods themselves cannot be compared, these indices cannot be compared? $\endgroup$ – James Steele May 2 '16 at 17:21
  • $\begingroup$ @JamesSteele: do they have citations for that? I get what they are trying to get at (basically a generalization of the $r^2$ statistic), but I've never read anything about such a metric and how well it works. However, that's a very different statement than "that metric does not work". $\endgroup$ – Cliff AB May 2 '16 at 17:26
  • $\begingroup$ the citation is Greene, W.H. (1997), Econometric Analysis, 3rd ed. but it appears that the 7th edition is out now, here is a pdf: rum.prf.jcu.cz/public/mecirova/eng_ekonomka/… I found on page 573 the reference to the likelihood ratio index "which has come to be known as the pseudo r square" $\endgroup$ – James Steele May 2 '16 at 17:32
  • 2
    $\begingroup$ Your basis of computing and comparing log-likelihoods appears invalid, because you have not included the implicit normalizing constants. Since they differ between the two models, they must be included. $\endgroup$ – whuber May 2 '16 at 20:15
  • 1
    $\begingroup$ @whuber: ah, that's correct. Too accustomed to ignoring for optimization. Fixed. $\endgroup$ – Cliff AB May 2 '16 at 20:30
-1
$\begingroup$

This is a complex task. The idea is to compare 2 different models which don’t have the input-output pairs and that is a big problem. Obviously you cannot compare log-likelihood directly however comparing log-likehood ratio index is not too bad. This metric is also called Deviance.

Actually, comparing the ratio of log-likelihood is as fair as comparing accuracy rate between the 2 models. So it is not too bad but as you mention it, not ok. How do you fairly compare 4 accuracies on one side (confusion matrix) and 9 on the other?...

Now this is close to the very common problem in ML of complexity VS accuracy. Complexity usually refers to the number of parameters to fit. The more complex your algorithm is, the better you can fit the training data. However it could lead to bad prediction power. A lot of work has been done with AIC, BIC and other criterion but they still differ from your problem. They compare internal model complexity and not number of response which could be seen as external model complexity.

If you want to look at this problem in the same way, you need to find a way to measure the cost of adding one response variable to your model, and then you can compare that with the log-likelihood ratio index for both models and tell which model is best along those 2 dimensions... Not easy, I haven't seen anything on that topic before and the reasons is that somehow, this is like comparing apples and oranges.

Then finally the question is should you do nothing ? Probably not, you try to do something as good as you can and thus you use the log-likelihood index.

Hope this help

$\endgroup$
  • $\begingroup$ Actually, most of what you claim is the opposite of what will happen. The more categories you add, the lower the log-likelihood, not the higher. And why do you think comparing ratios of log-likelihoods (note that this is not the ratio of fit1/fit2, but rather the ratios fit1/fit1Null to fit2/fit2Null)? Reflexively, it seems like this might be okay. However, it doesn't take too much exploration to see that it is very problematic (as the citation from Greene, mentioned in the comments, states as well). $\endgroup$ – Cliff AB May 2 '16 at 19:31
  • $\begingroup$ I cleaned and re-focused my post. $\endgroup$ – Romain May 2 '16 at 19:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.