Comparing logistic regression models with same number of parameters What would be an appropriate way to compare two logistic regression models with the same number of parameters (i.e., model 1 is not nested in model 2)? In my case, I cannot compare the models to a superset model with all predictors of both models, because including the differing parameters of both models results in nonidentifiability due to multicolliniearity.
 A: Although they are non-nested, the AIC/BIC or likelihood (in your special case) seem to make sense to compare which one is better fit.
Some study argue that models should be nested, so the DIC is more sophisticated and solid.
For it is a classifier, ROC/AUC or correct-rate are also a good way to compare those of them, in terms of the training performance or CV performance.
A: I would suggest you to use AIC or BIC, which are good for comparing non-nested models. You just have to make sure that:


*

*Both models have the same N if you are using BIC:
BIC adjusts for sample size: $BIC = -2\ln(L) + \ln(N)k$ with $N$ being sample size, $L$ being the likelihood, and $k$ being the number of parameters in the model. It penalises large models and large samples.
On the other hand, AIC adjusts only for number of parameters: $AIC = -2\ln(L) + 2k$.
Sometimes you may end up with different sample sizes for models because of differences in missing data for the variables you include. It is not a good idea to compare models that have different sample sizes, and BIC is specially sensitive to such differences.

*The likelihood being modelled refers to the same thing:
This seems to be the case because you use logistic regression for both models, but it may happen if you are comparing different types of model. For example, even though discrete-time logit models and parametric exponential-time survival regression are both types of event-history model, they differ in terms of what their likelihoods represent. The likelihood in the discrete-time logit is based on the number of failures in the risk set, while the likelihood in the exponential model (and in other parametric continuous-time event-history models) is based on the survival times in the data. Therefore, the likelihoods may differ in value just because of that, even if one model is not better than the other.
This thread at statalist and this help file address a similar question as yours.
A good addition to this answer as provided in the comments: if your model has the same number of parameters and sample size, you are effectively comparing the deviances of the two models by comparing BICs or AICs, what also works fine.
