I have several models that I want to ensemble together. I know that this will only help if the models are uncorrelated, e.g. they don't make the same mistakes. Is there a way to figure out if any two models are correlated without actually testing them individually using the data?
One method might be to calculate the residuals of each model and determine their correlation. If errors are correlated, then they fail in similar ways which I believe is what you're trying to avoid.
It really depends if it is a regression or a classification problem, if your speaking from a strict machine learning perspective. With that in mind, I would, for a classification problem:
- inspect the confusion matrix between models
- plot key statistics (recal, F1, precision, accuracy) by bin's of the key variables of the model
if it is a regression problem from an ml perspective I would do the same, but inspecting the r-squared, adjusted-r-squared, MSE, MAE and etc.
If your using Bayesian methods strictly, I would chill and Bayesian average all models based on their WAIC share.