4
$\begingroup$

New to machine learning and have been reading about ensemble modelling. A statement that keeps reappearing is: "Ensemble models work better when we ensemble models of low correlation."

I've seen examples, but I can't find an explanation for why this case? Could anyone shed some light?

$\endgroup$
3
$\begingroup$

You wake up in the morning and want to decide whether to take an umbrella with you when leaving. You look at the weather forecast from five newspapers on the assumption that their combined prediction would be more accurate than the prediction of only one of them. Does your assumption still hold if instead of looking at the prediction from five different newspapers (possibly using different weather models) you look at the predictions of five copies of the same newspaper?

| cite | improve this answer | |
$\endgroup$
3
$\begingroup$

It relates to a very popular word these days:

Conspiracy!

Think about how poor classifiers might hurt an ensemble that "votes" on the class to predict. They could add noise, sure, but as long as there are enough good classifiers to cast their votes in the right direction, the addition of the poor classifiers is unlikely to do much harm.

Unless, of course, the poor classifiers "team up." There's already too much personification already in this answer, so the point is that highly correlated poor classifiers could overturn the collective answer of the better ones.

Also, positively correlated data leads to less precise estimates in general, because there is less information than your sample size would suggest. For arithmetic means, you can see this in the variance formula for sums of correlated random variables.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

Models of different nature tend to deviate from the ground truth in very different directions, thus, promediating a closer-to-zero error. Models with high correlation tend to deviate towards same direction, thus, promediating a greater and more correlated error.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

Likelihood is that each learner is weak, meaning they have low accuracy. The reason the learners have low correlation is because they tend to be accurate at different times, because they have different "expertise" or "skills." The idea of ensembling is to combine their skills into one strong model.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.