What would be an appropriate way to compare two logistic regression models with the same number of parameters (i.e., model 1 is not nested in model 2)? In my case, I cannot compare the models to a superset model with all predictors of both models, because including the differing parameters of both models results in nonidentifiability due to multicolliniearity.
1
-
$\begingroup$ For non-nested model comparisons, you can use DeLong’s test, Vuong’s test, R^2 based test (sciencedirect.com/science/article/pii/S0002929723000046), which is available in CRAN (cran.r-project.org/web/packages/r2redux/index.html), and R^2 based AUC test (biorxiv.org/content/10.1101/2023.08.01.551571v1.full), which is available in CRAN (cran.r-project.org/web/packages/R2ROC/index.html). $\endgroup$– BiyongAug 24 at 1:15
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
Although they are non-nested, the AIC/BIC or likelihood (in your special case) seem to make sense to compare which one is better fit.
Some study argue that models should be nested, so the DIC is more sophisticated and solid.
For it is a classifier, ROC/AUC or correct-rate are also a good way to compare those of them, in terms of the training performance or CV performance.
-
1$\begingroup$ ROC or AUC ($c$-index) are not efficient ways to compare two models. This is like comparing two Wilcoxon tests instead of doing the correct head-to-head comparison. Proportional "classified" correctly is far worse. $\endgroup$ Aug 24, 2014 at 13:25
-
$\begingroup$ @FrankHarrell, Yep I agree with you that usually ROC/AUC can not clearly and efficiently rank models in practice cases. Just think of such indicators do not depend on any assumptions and have the same scale to compare, and easily to understand for outsiders. Do you have any other suggestion? Thanks. $\endgroup$– VincentAug 26, 2014 at 10:19
-
1$\begingroup$ There are several efficient approaches: Test a union of the two models against each model using a likelihood ratio test; compare generalized $R^2$; test whether one model is "more concordant" than the other using pairs of pairs of predictions (R
Hmiscpackagercorrp.censfunction). $\endgroup$ Aug 26, 2014 at 11:59
$\begingroup$
$\endgroup$
6
I would suggest you to use AIC or BIC, which are good for comparing non-nested models. You just have to make sure that:
Both models have the same N if you are using BIC:
BIC adjusts for sample size: $BIC = -2\ln(L) + \ln(N)k$ with $N$ being sample size, $L$ being the likelihood, and $k$ being the number of parameters in the model. It penalises large models and large samples.
On the other hand, AIC adjusts only for number of parameters: $AIC = -2\ln(L) + 2k$.
Sometimes you may end up with different sample sizes for models because of differences in missing data for the variables you include. It is not a good idea to compare models that have different sample sizes, and BIC is specially sensitive to such differences.
The likelihood being modelled refers to the same thing:
This seems to be the case because you use logistic regression for both models, but it may happen if you are comparing different types of model. For example, even though discrete-time logit models and parametric exponential-time survival regression are both types of event-history model, they differ in terms of what their likelihoods represent. The likelihood in the discrete-time logit is based on the number of failures in the risk set, while the likelihood in the exponential model (and in other parametric continuous-time event-history models) is based on the survival times in the data. Therefore, the likelihoods may differ in value just because of that, even if one model is not better than the other.
This thread at statalist and this help file address a similar question as yours.
A good addition to this answer as provided in the comments: if your model has the same number of parameters and sample size, you are effectively comparing the deviances of the two models by comparing BICs or AICs, what also works fine.
-
$\begingroup$ 1. Both models need the same N whatever you're using surely? Otherwise what is being compared to what? 2. I don't think 'likelihood' is what you mean here. If the likelihood functions are the same, then the models just are the same. $\endgroup$ Aug 24, 2014 at 14:32
-
1$\begingroup$ BIC requires N to be the same, otherwise BIC's will differ just by virtue of different N. As for "likelihood being modelled is the same," I meant that the two likelihoods should measure the same thing. For example, comparing a discrete-time logit model with an exponential survival model is not a good idea because the likelihoods of these two models refers to two different things – failures within the risk set in the first and survival times in the second. I edited the answer to make it clearer. $\endgroup$– KenjiAug 24, 2014 at 16:55
-
$\begingroup$ Is your likelihood point not covered by saying that the models should agree about what the observations are? $\endgroup$ Aug 24, 2014 at 17:27
-
2$\begingroup$ Note that when $k$ & $N$ are the same, comparing the AIC & BIC amount to simply comparing the deviances (which is valid in this case). $\endgroup$ Aug 24, 2014 at 17:33
-
$\begingroup$ @conjugateprior Not necessarily. In both types of survival model, the observations will consist of multiple individual records of survival times and a variable indicating failure at that time period. Data for discrete-time and continuous event history may be different, though: you don't need multiple records for continuous time. A better example can be Cox vs Exponential regression: you can use the exact same data and observations, and yet the likelihoods will differ. $\endgroup$– KenjiAug 24, 2014 at 17:33