In general the xgboost classifier is built by the idea of reducing the total error. Im using both xgboost and random forest to classify using small dataset (181 observations) and i noticed that the random forest outperformed xgboost accuracy with test data while xgboost gave higher accuracy with train data and i can't explain why? worth to mention i have divided the data to 60:40% and i have low accuracy in general with all classifiers, and SVM was the lowest among all.

  • 1
    $\begingroup$ Sounds like a classic case of overfitting $\endgroup$
    – Sycorax
    Commented Apr 29, 2020 at 15:15
  • $\begingroup$ okay i understand it is overfitting. but i wanted to know if there is any theory behind this result when we expect xgboost to perform better than random forest while it is not in some cases apparently. $\endgroup$
    – Jude83
    Commented Apr 29, 2020 at 16:00
  • $\begingroup$ Powerful models are also prone to overfitting; this is formalized in the bias-variance-tradeoff. Using a powerful model with a small data set is also especially at risk of overfitting. $\endgroup$
    – Sycorax
    Commented Apr 29, 2020 at 16:19
  • $\begingroup$ Thank you for your reply. $\endgroup$
    – Jude83
    Commented Apr 29, 2020 at 16:57

1 Answer 1


While using classifiers, setting the value of parameters specific to particular classifier impact it's performance.

Check the number of estimators, regularisation cofficient e.t.c. In your case Xgboost model is suffering from overfitting problem.

RandomForest is less prone to overfitting as compared to Xgboost. This is due to their underlying principles on which both work.

To reduce overfitting in your Xgboost model, reduce number of estimators, increase the regularisation cofficient value.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.