I am learning about boosting and bagging from the Amazon Web Services Machine Learning courses. In it, they describe bagging and boosting as ways to automate feature extraction and selection.

My understaning is that boosting is a method by which you have several weak models trained in sequence. Each is trained on the full trainig data, but with greater emphasis placed on the weaknesses of the previously trained model.

In contrast, bagging is where you take a collection of weaker models and train them each on a subset of the features. To make a prediction in either bagging or boosting, combine the results of the weak models by (weighted) averaging or voting.

What I don't understand, is how either of these can be considered to be selecting features. It seems to me that the feature sets for the different sub-models are chosen essentially randomly, which means that there's no compression or exclusion of variables as in, for instance, Principal Component Analysis.

Are bagging and boosting methods of extracting and selecting features?

If so, what am I missing about them, or about the meaning of extraction/selection?

Or was this description by the speaker misleading?

This states that these kinds of methods extract feature importances, and that the more decisions (in the case of a decision tree) based on that feature, the greater a feature's importance. However, I don't understand if/how this helps to speed the training process, as I understand the purpose of extraction/selection to be. As a programmer, I could use that information to exclude unhelpful features on subsequent trainings, but it doesn't quite fit my perception of what 'automated feature selection' would be.

This also provides some information on the subject, but I had a difficult time reading it. Again, it described that it could be used for feature selection, but it seems to me similar in role (not method) to something more mundane like PCA.

Thanks in advance!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.