I am looking to know what to do with bagging when n (observations) << p (features), or we have wide data. Note that each of the features are useful/significant/required. So I cannot subset my features.

Bagging, as it stands, addresses the problem of high variance in decision trees by creating multiple trees from bootstrapped observations. So when n << p, we really do not care whether each individual tree overfits, right? So bagging should work just fine with wide data. Or is there something we need to do, be careful about?

Appreciate your discussion on this idea.

This is not a textbook question


Bagging would help stabilize predictions, so it's worth a try, but you'll have to ensure models in the ensemble don't end all looking the same (high correlation between models is bad for bagging).

So you could employ another concept, attribute bagging (otherwise called "Random subspace method") is useful in that scenario, I myself employed it.

It's an ensemble method analogous to bagging, where you subset features instead of samples (Random Forests employ both at once, for example).

Check this paper [1] that details the procedure and the properties.

From personal experience, AB can work very well, even without bagging in $p \gg n$ problems.

[1] Bryll, R., Gutierrez-Osuna, R., & Quek, F. (2003). Attribute bagging: improving accuracy of classifier ensembles by using random feature subsets. Pattern recognition, 36(6), 1291-1302.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.