Suppose I provide predictive models estimating the likelihood someone (for the sake of argument, say recent college graduates only) defaults on a mortgage loan. One of my customers, before having had a chance to truly use my predictions in their day-to-day work, wants to compare my model with one of theirs. Their model is intended to serve the same purpose, applied to the same cohort of individuals/loan type (i.e. mortgages).
It's highly likely the methodology may differ between our two approaches, even though the outcome we're predicting is the same – whether someone defaulted. I may have collected data only between 2015-2017, while they collected from 2017-2018 only. The event rates are also likely to differ, although that may not matter as much in a test of how well each model rank-orders a population, separately. You could imagine all of the slight ways in which the actual datasets modeled off of can differ between these two models.
What are some effective and statistically valid ways of comparing the performance of these two models, without having the opportunity at this point to run an experiment pitting the two models against each other in real-time? Would it be a fair comparison to simply prefer the model with the greater AUROC (ignoring other potentially more useful performance metrics for the time being)? The ultimate goal of the model is to rank-order these college graduates, so a statistic like AUROC seems appropriate.