2
$\begingroup$

I am referring to this link to Feature Transformation using tree ensembles for the context.

Specifically for below part of code, in the sample of the link, the method of (1) using Boosting tree to generate feature, then using LR to train, outperforms (2) using Boosting tree itself. Questions,

  1. Wondering if it is true in general case using Boosting tree to generate feature (and using another classifier to classify) is better than using Boosting tree to do classification itself?
  2. And also wondering why using Boosting tree to generate feature, then using LR to train, outperforms using Boosting tree itself?

    grd = GradientBoostingClassifier(n_estimators=n_estimator)
    grd_enc = OneHotEncoder()
    grd_lm = LogisticRegression()
    grd.fit(X_train, y_train)
    grd_enc.fit(grd.apply(X_train)[:, :, 0])
    grd_lm.fit(grd_enc.transform(grd.apply(X_train_lr)[:, :, 0]), y_train_lr)
    
$\endgroup$
3
+25
$\begingroup$

The better performance of GBT+LR in the mentioned example is just because larger training set than GBT. Please note the GBT use X_train for training, while GBT+LR use X_train and X_train_lr. I prepared a new classificator GBT-ext, that is learned on the full training set, same as the GBT+LR.

grd_ext = GradientBoostingClassifier(n_estimators=n_estimator)
grd_ext.fit(np.concatenate([X_train, X_train_lr]), np.concatenate([y_train, 
y_train_lr]))

y_pred_grd_ext = grd_ext.predict_proba(X_test)[:, 1]
fpr_grd_ext, tpr_grd_ext, _ = roc_curve(y_test, y_pred_grd_ext)

Results of the GBT-EXT are almost the same as the GBT+LR. See the picture below: enter image description here

I think the result above answers both of your questions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.