Methods for comparing two different models on similar problem domains

Question

Suppose I provide predictive models estimating the likelihood someone (for the sake of argument, say recent college graduates only) defaults on a mortgage loan. One of my customers, before having had a chance to truly use my predictions in their day-to-day work, wants to compare my model with one of theirs. Their model is intended to serve the same purpose, applied to the same cohort of individuals/loan type (i.e. mortgages).

It's highly likely the methodology may differ between our two approaches, even though the outcome we're predicting is the same – whether someone defaulted. I may have collected data only between 2015-2017, while they collected from 2017-2018 only. The event rates are also likely to differ, although that may not matter as much in a test of how well each model rank-orders a population, separately. You could imagine all of the slight ways in which the actual datasets modeled off of can differ between these two models.

What are some effective and statistically valid ways of comparing the performance of these two models, without having the opportunity at this point to run an experiment pitting the two models against each other in real-time? Would it be a fair comparison to simply prefer the model with the greater AUROC (ignoring other potentially more useful performance metrics for the time being)? The ultimate goal of the model is to rank-order these college graduates, so a statistic like AUROC seems appropriate.

Answer 1

First of all regarding using ROCAUC, it is criticised heavily for not being easy to interpret, prone to class imbalance and famously not being a coherent measure and that it can not be used to compare two classifiers, see h-measure. Interpreting the result of ROCAUC should be practiced with strong caution. So in this case, most probably both models have this differences h-measure refers to. If both parties access to h-measure, that would be a statistically sound way to compare.

However, you would need to run models against each other for the most reliable comparison with a matching test set if you have access to both data sets, while comparing two performances conceptually the same thing: a matching test set set that is representative of both model's test set as much as possible, i.e., similarity wise. This can be achieved using some sort of matching algorithm, similar to ones used to create matching sets in propensity score matching.