# Using bagging and random forests together

I was looking at a kernel implementation (for text classification) and the following piece of code got me a little bit confused (I removed part of the features - in order to keep it light - as most of them are similar - e.g. number of negative, positive, neutral words):

pipeline_ = Pipeline([
('fu', FeatureUnion([
('tfdif_features', Pipeline([
('cv', CountVectorizer()),
('tfidf', TfidfTransformer()),
('tfidf_', Wrapper(RandomForestClassifier())),
])),
('nb_pos_features', Pipeline([
('nb_pos', NumberSelector('posWords') ),
('nb_pos_', Wrapper(RandomForestClassifier())),
])),
])),
('xgb', XGBClassifier()),
])


At first I thought that it looks like bagging, as for each feature a base model is created, followed by boosting. But isn't the Random Forest in this case a Decision Tree? Why using Random Forest and not something else (e.g. Logistic regression)?

Any clarification will be greatly appreciated.

• Welcome to CV. I think you are asking about the difference between a CART, a logistic regression, a RF, and a sequential learner comprised of two random forests that input into a gradient boosted tree. – EngrStudent Jun 20 '19 at 13:02

But isn't the Random Forest in this case a Decision Tree?

All random forest models are ensembles of decision trees. Random forest constructs many decision trees using a randomization procedure.

Why using Random Forest and not something else (e.g. Logistic regression)?

Because the author of this code made a deliberate choice. You'll have to ask the author why they chose Random Forest instead of a different method. Generically, tree induction methods like random forest are very flexible, whereas logistic regression is merely a linear model.

• Thank you for your answer! – moz_szt Jun 21 '19 at 8:14
• @moz_szt if you’ve found my answer helpful, please consider upvoting and/or accepting it. More information about how the site works can be found in the help center. – Sycorax Jun 21 '19 at 13:25

What the model is intending to do is use the output of RandomForest1 and RandomForest2 as features into the XGBoost Classifier

• Thank you for your answer. I understood that part, but I didn't quite understood why using a Random Forest for only one feature? Why not using Logistic Regression or something else? – moz_szt Jun 19 '19 at 13:59
• @moz_szt, the TFIDF vectorizer will translate each row to multiple features. It is not a single feature. – Anant Gupta Jun 19 '19 at 17:03