Whether can stacked generalization (stacking) further improve the performance if learner A significantly outperform B?

Question

We know that stacking is the most popular meta-learning technique. It learns from the predictions of the base learners that learn from the training dataset. Now assuming there are two base learners, denoted as A and B, they have been trained on the training dataset training dataset $D_{train}$. Learner A outperform distinctly B on the validation dataset $D_{validation}$, as indicated by the ROC curve totally enclosing that of B on $D_{validation}$, as illustrated by the following figure. Under such case, can stacking of A and B further improve the performance on $D_{validation}$?

I encountered this question when dealing with my data in geosciences, which indicated the stacked meta-learner can only achieve as most good result as that of A. Since the data is in private, I cannot share the project.

This figure is just for illustration, which is cited from:https://classeval.wordpress.com/introduction/introduction-to-the-roc-receiver-operating-characteristics-plot/

Answer 1

I must admit I am not awfully experienced with stacking, however I've done a good amount of reading so here's my two cents.

To be clear: by "Under such case, can stacking of A and B further improve the performance", I assume you mean you're training some model C on the outputs of both A and B.

My response:

You have two base learners, one of which is undoubtedly superior to other overall. However, a meta-learner trained on both, A and B, could still outperform A (the top base performer).

The reason is that if A and B are different (different algorithms, different hyperparameters, trained on different rows/features etc.), they could have different biases. So even though on average A is better, it could be the case that it's not flat out better in every scenario. Imagine you have a dataset of some measurement made in males (20% of data) and females (80% of data). A could be much better than B on average, but let's say B is really good at the male rows. This situation-specific superiority of B would likely be drowned out when looking at the overall performance like you are, BUT a meta-learner could be able to pick up on the fact that when male, B tends to be more accurate but broadly A is better.

My point here, is that even though one model could be better overall, there is the possibility that the models A and B have biases different enough that a meta-learner could discern these orthogonal biases and learn to rely more on one than the other in certain situations (leading to perhaps only a very marginal improvement in performance).

However, if the biases/errors between A and B are not very different, then you almost certainly won't get performance improvement by stacking. For example, if A is a tuned RF and B is an unoptimized RF trained on some random subset of the data (there's no reason at ALL to think that B would have any unique insight that A doesn't have), then there's no reason stacking would help.