Test to select best models in production I've got four models in production and using the average of them as the served prediction. We get ground truth data immediately.
I've optimized them and found the best models during my training/testing phase deployed all four of them, then took the average and served that as the prediction.
Out of these four models two seem to be performing far better than the others and I want  a reasonable justification for model selection when the models are already tuned and deployed.
I can also test some hypothesis and do statistics to assess each models quality.
Would anyone object - ie, would you point and laugh at me - if I used AIC to perform model selection here? To narrow down from these four models to one?
 A: Model selection criteria, such as the BIC, the AIC, or the minimum length criterion, are commonly used in the literature, so nobody would laugh and point at you if you use them (and if they do, please reconsider your acquaintance).
However, the validity of these criteria rely on some strong assumptions, that you will need to verify and justify. For instance, using the BIC requires that your data are i.i.d., that you have enough of them, that you correctly obtained your Maximum Likelihood Estimator, and that you can non ambiguously compute the number of free parameters in your models, which are not always obvious.
Another possibility is to perform Bayesian Model Averaging, in which you use the average of the output of different models, but weighted by their respective uncertainty: https://www.jstor.org/stable/2676803?seq=1#metadata_info_tab_contents
Some references:

*

*Burnham, K. P., & Anderson, D. R. (2004). Multimodel inference: understanding AIC and BIC in model selection. Sociological methods & research, 33(2), 261-304.

*Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & Van Der Linde, A. (2002). Bayesian measures of model complexity and fit. Journal of the royal statistical society: Series b (statistical methodology), 64(4), 583-639.

