0
$\begingroup$

I have several models that I want to ensemble together. I know that this will only help if the models are uncorrelated, e.g. they don't make the same mistakes. Is there a way to figure out if any two models are correlated without actually testing them individually using the data?

$\endgroup$
  • $\begingroup$ What models are you using? If you are combining models that have an element of randomness (e.g. RF) with something like SVMs you should expect those models to not be particularly correlated. Combining deterministic with random models should help you avoid correlation. $\endgroup$ – francium87d Jun 14 '17 at 23:12
  • $\begingroup$ Thanks that makes sense. I was wondering if there exists a model selection criterion that could identify the correlation between two models. Just trying to save some time. $\endgroup$ – Kevin Jun 14 '17 at 23:46
1
$\begingroup$

One method might be to calculate the residuals of each model and determine their correlation. If errors are correlated, then they fail in similar ways which I believe is what you're trying to avoid.

$\endgroup$
0
$\begingroup$

It really depends if it is a regression or a classification problem, if your speaking from a strict machine learning perspective. With that in mind, I would, for a classification problem:

  • inspect the confusion matrix between models
  • plot key statistics (recal, F1, precision, accuracy) by bin's of the key variables of the model

if it is a regression problem from an ml perspective I would do the same, but inspecting the r-squared, adjusted-r-squared, MSE, MAE and etc.

If your using Bayesian methods strictly, I would chill and Bayesian average all models based on their WAIC share.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.