I've got four models in production and using the average of them as the served prediction. We get ground truth data immediately.
I've optimized them and found the best models during my training/testing phase deployed all four of them, then took the average and served that as the prediction.
Out of these four models two seem to be performing far better than the others and I want a reasonable justification for model selection when the models are already tuned and deployed.
I can also test some hypothesis and do statistics to assess each models quality. Would anyone object - ie, would you point and laugh at me - if I used AIC to perform model selection here? To narrow down from these four models to one?