4
$\begingroup$

When using stacking, can accuracy always be improved by adding more base classifiers, types of base classifiers, and features?

$\endgroup$
4
$\begingroup$

As with any classifier, adding new input features can improve classification accuracy when when the new features contain new information about the labels. This performance improvement isn't guaranteed because classifiers are imperfect and may not be able to exploit the information. If the new features share information with existing features, the new features may or may not help. For example, multiple noisy copies (or invertibly transformed versions) of a signal can help to 'average out' the noise. But, if there's no noise or the noise is correlated across signals, then the multiple copies may just be redundant. If the new features don't contain information about the labels, they'll do nothing in the best case and hurt performance in the worst case. In all cases, the curse of dimensionality must be considered. The presence of more features can hurt many classifiers and can increase opportunities for overfitting. It's possible that this effect could overshadow the benefit of new features if that benefit is small.

The situation is similar when adding new base classifiers to a stacking setup, because the base classifiers' outputs are features for the final classifier. All the same arguments from above hold here. In this case, these 'second level' features are likely correlated because all base classifiers are all trying to predict the same thing. But, they do it suboptimally. The hope is that they behave in different ways, so that the final classifier can combine the noisy predictions into a better final prediction. Loosely, then, adding new base classifiers has the best chance of helping when they do a good job and behave differently than existing base classifiers, but this isn't guaranteed. If the new classifiers perform at chance they can't help, and will probably hurt. The final classifier can overfit, and providing it with more base classifiers may increase its ability to do so.

$\endgroup$
  • $\begingroup$ This is exactly the sort of explanation I needed. I have already accepted as best answer, but I just wanted to say thanks. $\endgroup$ – mino Jul 14 '16 at 20:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.