When should I not use an ensemble classifier?

Question

In general, in a classification problem where the goal is to accurately predict out-of-sample class membership, when should I not to use an ensemble classifier?

This question is closely related to Why not always use ensemble learning?. That question asks why we don't use ensembles all the time. I want to know if there are cases in which ensembles are known to be worse (not just "not better and a waste of time") than a non-ensemble equivalent.

And by "ensemble classifier" I'm specifically referring to classifiers like AdaBoost and random forests, as opposed to, e.g., a roll-your-own boosted support vector machine.

Answer 1

The model that is closest to the true data generating process will always be best and will beat most ensemble methods. So if the data come from a linear process lm() will be much superior to random forests, e.g.:

    set.seed(1234)
p=10
N=1000
#covariates
x = matrix(rnorm(N*p),ncol=p)
#coefficients:
b = round(rnorm(p),2)
y = x %*% b + rnorm(N)
train=sample(N, N/2)
data = cbind.data.frame(y,x)
colnames(data) = c("y", paste0("x",1:p))
#linear model
fit1 = lm(y ~ ., data = data[train,])
summary(fit1)
yPred1 =predict(fit1,data[-train,])
round(mean(abs(yPred1-data[-train,"y"])),2)#0.79

library(randomForest)
fit2 = randomForest(y ~ ., data = data[train,],ntree=1000)
yPred2 =predict(fit2,data[-train,])
round(mean(abs(yPred2-data[-train,"y"])),2)#1.33

Answer 2

I do not recommend using an ensemble classifier when your model needs to be interpretable and explainable. Sometimes you need predictions and explanations of the predictions.

When you need to convince people that the predictions are worth believing, a highly accurate model can be very persuasive, but I have struggled to convince people to act on predictions when the methods are too complex for their comfort level.

In my experience, most people are comfortable with linear additive models, models they could score by hand, and if you try to explain adaptive boosting, hyper-planes and 5th level interaction effects they will respond as if you are pitching them black magic.

On the other hand, people can be comfortable with the complexity of the model, but still want to internalize some insight. Scientists, for example, might not consider a black-box model to be an advance in human knowledge, even if the model is highly accurate.

Variable importance analysis can help with insights, but if the ensemble is more accurate than a linear additive model, the ensemble is probably exploiting some non-linear and interaction effects that the variable importance analysis can't completely account for.

Answer 3

I would like to add to branco's answer. The ensembles can be highly competitive and provide very good results. In academics for example, this is what counts. In industry, the ensembles may be too difficult to implement/maintain/modify/port. Goef Hinton's work on "Dark Knowledge" is exactly about this: how to transfer the "knowledge" of a large ensemble into one easy to move around model. He states that ensembles are bad at test time: they are highly redundant and the computation time can be of concern.

His team got some interesting results, I suggest to check out his publications or at least the slides. If my memory is good, this was one of 2013 or 2014 hot topics.

The slides about Dark Knowledge can be found here: http://www.ttic.edu/dl/dark14.pdf

Answer 4

In short, ensemble methods are good if your model needs to learn complex patterns in the data. In other cases, ensemble methods are prone to over-complex the solution and do not have a robust model. For instance, in the case of having outliers in your regression problem or having linear features in your classifier problem.