I am practicing classification with machine learning on a very large set of samples (about 20,000), where half of which are labelled training data and the other half are the testing data. There are 13 categories of features with a classification of 9 different families. I have read up on different models to use to get the best accuracy of prediction, but I'm stuck at this point and don't know which order to do things.

I have my feature matrix of both the training and testing data and would like to start training my model. From what I understand, bagging is very popular in problems like this. And I would like to test out a Logistic Regression model as well as XGBoost.

Do I use bagging before testing my models or do I train my models and then use bagging to reduce the variance? What is the relationship between the bagging and the two models?


2 Answers 2


Bagging works as follows: You resample your training data $m$ times, create $m$ estimators, fit each estimator to a resampled training set, and then average their predictions over new data.

Logistic regression is not a method which uses bagging, but you can bag a bunch of logistic regressions.

The "GB" in XGBoost stands for "Gradient Boosting" which is a different technique.


In my master thesis I used (among others) a Random Forest, which also uses the bagging technique. Below the part from my thesis related to bagging. You have different models than I had, but the idea is the same. You take subsamples of your training data and fit your model on these subsamples. Each model makes a certain classification per observation in your testing data. Lastly, you make the final classification per observation by checking which class got the most 'votes'.

So to answer your questions:

-You train your models and then use bagging

-There is no real relation between bagging and your models. Bagging is an estimation technique (AFAIK) which can be applied to all type of models.

Lastly, I think that XGBoost incorporates bagging automatically (not sure though), so don't forget to check the documentation.

Thesis part:

Bagging is the application of the bootstrap method applied to high-variance machine learning algorithms. Bagging uses multiple machine learning algorithms in order to improve the accuracy and stability of the individual algorithms. Furthermore, it reduces variance and the probability of overfitting. The CART algorithm for example has a high variance due to the fact that it depends heavily on the training set that is used. A couple of extra outliers in the training sample can change the whole tree. The bagging procedure applied to the CART algorithm is as follows:

1) Create $B$ sub-samples;

2) Fit a CART tree on each different sub-sample to obtain the ensemble of trees $\{CART_b\}^B_1$;

3)Use each individual tree to make a prediction;

4)The final prediction is obtained by taking the average prediction (for regression trees) or to take the value that is predicted the most (for classification trees).

By using many different trees, the effects of outliers will be averaged out and hence, the predictions are less volatile than the predictions of a single CART tree.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.