We know that stacking is the most popular meta-learning technique. It learns from the predictions of the base learners that learn from the training dataset. Now assuming there are two base learners, denoted as A and B, they have been trained on the training dataset training dataset $D_{train}$. Learner A outperform distinctly B on the validation dataset $D_{validation}$, as indicated by the ROC curve totally enclosing that of B on $D_{validation}$, as illustrated by the following figure. Under such case, can stacking of A and B further improve the performance on $D_{validation}$?

I encountered this question when dealing with my data in geosciences, which indicated the stacked meta-learner can only achieve as most good result as that of A. Since the data is in private, I cannot share the project.

enter image description here

This figure is just for illustration, which is cited from:https://classeval.wordpress.com/introduction/introduction-to-the-roc-receiver-operating-characteristics-plot/


1 Answer 1


I must admit I am not awfully experienced with stacking, however I've done a good amount of reading so here's my two cents.

To be clear: by "Under such case, can stacking of A and B further improve the performance", I assume you mean you're training some model C on the outputs of both A and B.

My response:

You have two base learners, one of which is undoubtedly superior to other overall. However, a meta-learner trained on both, A and B, could still outperform A (the top base performer).

The reason is that if A and B are different (different algorithms, different hyperparameters, trained on different rows/features etc.), they could have different biases. So even though on average A is better, it could be the case that it's not flat out better in every scenario. Imagine you have a dataset of some measurement made in males (20% of data) and females (80% of data). A could be much better than B on average, but let's say B is really good at the male rows. This situation-specific superiority of B would likely be drowned out when looking at the overall performance like you are, BUT a meta-learner could be able to pick up on the fact that when male, B tends to be more accurate but broadly A is better.

My point here, is that even though one model could be better overall, there is the possibility that the models A and B have biases different enough that a meta-learner could discern these orthogonal biases and learn to rely more on one than the other in certain situations (leading to perhaps only a very marginal improvement in performance).

However, if the biases/errors between A and B are not very different, then you almost certainly won't get performance improvement by stacking. For example, if A is a tuned RF and B is an unoptimized RF trained on some random subset of the data (there's no reason at ALL to think that B would have any unique insight that A doesn't have), then there's no reason stacking would help.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.