Is it correct to compare likelihood ratio indices between logistic regression and multinomial logistic regression models? In the paper "Including Transfer-Out Behavior in Retention Models: Using the NSLC Enrollment Search Data" 
(http://www.studentclearinghouse.org/colleges/files/ST_UofMD_casestudy.pdf) 
the author compares two models of student retention:


*

*A binary logistic regression (with DV coded as either retained/not retained)

*A multinomial logistic regression (with DV coded as retained/not retained/transferred out)


The author uses two criteria for model comparison (p.14):


*

*"Predictive ability" measured by the the log likelihood and model chi-square

*"Explanatory power" which is a comparison of which IVs were predictive in each model, and the change in probability associated with a change in each IV. 


Based on the goal of the study I understand why criteria #2 is useful for comparison. However, I am wondering if it is correct to compare likelihood ratios between these two types of models. Additionally, are there any other tests that could accurately compare logistic vs. multinomial models?
 A: Criteria 1 does not make any sense in this case. That is because these likelihoods are simply not comparable.
In particular, let's simplify this significantly to make this more clear. Let's forget about explanatory variables and pretend that we have 300 observations. In those 300 observations, 100 are retained, 100 are not retained and 100 transfer out. 
In the binary model, we would say 100 are retained and 200 are not retained. This would give us a log-likelihood of
$100 \times \log(1/3) + 200 \times \log(2/3) + \log \left( \frac {300!}{200!100!} \right) = -3.02$
(where did we get these numbers? We have 100 retained subjects who's probability of being in that category is 1/3, and 200 not-retained subjects who's probability of being in that category is 2/3. The final term is the normalizing constant for the binomial distribution: thanks to whuber for pointing that out).
In the multinomial model, we would have log-likelihood
$100 \times \log(1/3) + 100 \times \log(1/3) + 100 \times \log(1/3) 
+ \log \left( \frac{300!}{100!200!} \right) + \log \left( \frac {200!}{100!100!} \right)= -5.90$
So if we look at the log-likelihood to compare models, we see that the binary model is "favored"...even though it clearly contains less information than the multinomial model! If you want an even more extreme example, what if we had (very boringly) used only one group? Then we would have a log-likelihood of 0!
A: This is a complex task. The idea is to compare 2 different models which don’t have the input-output pairs and that is a big problem. 
Obviously you cannot compare log-likelihood directly however comparing log-likehood ratio index is not too bad. This metric is also called Deviance.
Actually, comparing the ratio of log-likelihood is as fair as comparing accuracy rate between the 2 models. So it is not too bad but as you mention it, not ok. How do you fairly compare 4 accuracies on one side (confusion matrix) and 9 on the other?... 
Now this is close to the very common problem in ML of complexity VS accuracy. Complexity usually refers to the number of parameters to fit. The more complex your algorithm is, the better you can fit the training data. However it could lead to bad prediction power. A lot of work has been done with AIC, BIC and other criterion but they still differ from your problem. They compare internal model complexity and not number of response which could be seen as external model complexity. 
If you want to look at this problem in the same way, you need to find a way to measure the cost of adding one response variable to your model, and then you can compare that with the log-likelihood ratio index for both models and tell which model is best along those 2 dimensions... Not easy, I haven't seen anything on that topic before and the reasons is that somehow, this is like comparing apples and oranges.
Then finally the question is should you do nothing ? Probably not, you try to do something as good as you can and thus you use the log-likelihood index.
Hope this help
