17
$\begingroup$

In general, in a classification problem where the goal is to accurately predict out-of-sample class membership, when should I not to use an ensemble classifier?

This question is closely related to Why not always use ensemble learning?. That question asks why we don't use ensembles all the time. I want to know if there are cases in which ensembles are known to be worse (not just "not better and a waste of time") than a non-ensemble equivalent.

And by "ensemble classifier" I'm specifically referring to classifiers like AdaBoost and random forests, as opposed to, e.g., a roll-your-own boosted support vector machine.

$\endgroup$
  • 2
    $\begingroup$ I would not use ensemble methods if you do not have diversity among individual methods. In other words, ensemble is useful when you combine diverse set of methods. $\endgroup$ – forecaster Jun 24 '15 at 0:57
  • 2
    $\begingroup$ @forecaster I could not help to add very good paper about what authors call "good" and "bad" diversity pages.bangor.ac.uk/~mas00a/papers/gblkMCS10.pdf in the context of ensembles $\endgroup$ – Vladislavs Dovgalecs Jul 1 '15 at 21:42
  • $\begingroup$ @xeon nice article. regardless of machine learning methods be it random forest or ensemble (combining) different methods, diversity definitely helps. There is strong theory behind this and it is called $nature$ and $biologically \ inspired$. $\endgroup$ – forecaster Jul 2 '15 at 0:36
7
$\begingroup$

The model that is closest to the true data generating process will always be best and will beat most ensemble methods. So if the data come from a linear process lm() will be much superior to random forests, e.g.:

    set.seed(1234)
p=10
N=1000
#covariates
x = matrix(rnorm(N*p),ncol=p)
#coefficients:
b = round(rnorm(p),2)
y = x %*% b + rnorm(N)
train=sample(N, N/2)
data = cbind.data.frame(y,x)
colnames(data) = c("y", paste0("x",1:p))
#linear model
fit1 = lm(y ~ ., data = data[train,])
summary(fit1)
yPred1 =predict(fit1,data[-train,])
round(mean(abs(yPred1-data[-train,"y"])),2)#0.79

library(randomForest)
fit2 = randomForest(y ~ ., data = data[train,],ntree=1000)
yPred2 =predict(fit2,data[-train,])
round(mean(abs(yPred2-data[-train,"y"])),2)#1.33
$\endgroup$
13
$\begingroup$

I do not recommend using an ensemble classifier when your model needs to be interpretable and explainable. Sometimes you need predictions and explanations of the predictions.

When you need to convince people that the predictions are worth believing, a highly accurate model can be very persuasive, but I have struggled to convince people to act on predictions when the methods are too complex for their comfort level.

In my experience, most people are comfortable with linear additive models, models they could score by hand, and if you try to explain adaptive boosting, hyper-planes and 5th level interaction effects they will respond as if you are pitching them black magic.

On the other hand, people can be comfortable with the complexity of the model, but still want to internalize some insight. Scientists, for example, might not consider a black-box model to be an advance in human knowledge, even if the model is highly accurate.

Variable importance analysis can help with insights, but if the ensemble is more accurate than a linear additive model, the ensemble is probably exploiting some non-linear and interaction effects that the variable importance analysis can't completely account for.

$\endgroup$
  • $\begingroup$ Not what I was after, but good points. +1 $\endgroup$ – shadowtalker Jun 25 '15 at 21:28
3
$\begingroup$

I would like to add to branco's answer. The ensembles can be highly competitive and provide very good results. In academics for example, this is what counts. In industry, the ensembles may be too difficult to implement/maintain/modify/port. Goef Hinton's work on "Dark Knowledge" is exactly about this: how to transfer the "knowledge" of a large ensemble into one easy to move around model. He states that ensembles are bad at test time: they are highly redundant and the computation time can be of concern.

His team got some interesting results, I suggest to check out his publications or at least the slides. If my memory is good, this was one of 2013 or 2014 hot topics.

The slides about Dark Knowledge can be found here: http://www.ttic.edu/dl/dark14.pdf

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.