5
$\begingroup$

I'm a bit perplexed after working on a kaggle data science competition for a month now. It's tabular data, all real/floating point numbers, on the order of maybe 100k examples, 100ish features, and a single output (real) number to predict. I've been using dense neural networks and have gotten good but middling results.

(edit - to give some context, most naive models get 30% error rate, trying hard with the NNs I get down to maybe 24-25% error rate, but then the best scores are all hitting 19-20%; and we have similar features.)

However consulting with people doing really well, they're all using things like XGBoost, LightGBM, etc. Looking into what these are, I see they are decision trees being optimized. I am confused why these work so well, or at least better than neural networks in certain contexts.

Can someone provide some context or background info for why these 'boosted decision tree' methods seem to be the state of the art for these types of data competitions? Would appreciate some insight, thanks!

Edit 2 - as a follow up/related question, could one 'boost' neural networks by fitting residuals from previous NNs? Do people do this/why don't they? I guess I'm just overall confused at trees being used so successfully for data regression, I don't understand how they could do so well vs. seemingly more numerically powerful and flexible algorithms.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Boosting is usually performed with weak learners (barely better than chance in most cases). Neural networks are most often strong learners (very often interpolating the training data), so there is not much benefit in boosting it. $\endgroup$
    – Firebug
    Aug 10, 2021 at 21:44
  • 1
    $\begingroup$ Could you expand on this idea? It seems odd at first glance: if a bunch of weak learners put together create a good prediction, why wouldn't a bunch of strong(er) learners create an even better prediction? :) $\endgroup$
    – JDS
    Aug 10, 2021 at 22:03
  • 1
    $\begingroup$ they do, but not through boosting. Bagging is an ensembling procedure that makes strong learners generalize better (a la Random Forests with unpruned decision trees). Boosting is about controlling bias, while bagging is about controlling variance. $\endgroup$
    – Firebug
    Aug 10, 2021 at 22:32

2 Answers 2

Reset to default
4
$\begingroup$

Here is my hand-wavey attempt for why those models perform unusually well in Kaggle and other data science competitions.

Trees can handle a ton of variables (even highly correlated ones) quite well because of the procedure in which they access them. A (CART) Tree won't use all variables to fit the next split but instead pick the next best one to create the decision. This allows us to throw everything in a data science competition at a tree-based model and quickly see what sticks. This is unlike a linear regression which will fit all parameters simultaneously so special care is needed when using a lot of variables.

Trees will model feature interactions naturally and efficiently. This makes trees really powerful in competitions because we usually have no domain knowledge or the variables themselves have been encoded. For linear regression we usually have to add dimensions via interaction terms or polynomial expansions in order to inject that knowledge but a tree with a depth of more than one is already doing it. Similarly for neural nets we have to add more layers which increases the parameters bigly.

So, trees are great but why not a random forest or some other ensembling technique rather than boosting? No idea, but boosting is great and specifically xgboost and lightgbm implementations use regularized trees along with algos which help speed up computation (like histogram splits) to deal with boosting's biggest computational issue: it is sequentially learned. And a lot of their development have been influenced greatly by kaggle and other data science competitions.

For your edit yes you could boost any algorithm since it is just an ensembling procedure but you do not get similar gains with algorithms like a NN or linear regression as you would with trees. This is because adding two trees adds complexity where as adding two sets of coefficients in linear regression leaves you with the same linear regression formula. Neural nets also can overfit so it is not a good candidate for boosting but there are are still relevant algos such as the resnet architecture which is theoretically related to boosting, see here.

Improve this answer
$\endgroup$
2
  • $\begingroup$ A point to add. "Trees will model feature interactions naturally and efficiently" - not exactly. There are kinds of interactions that are handled not so well by trees. For examples, when the target is decomposed additively or multiplicatively: $Y=f_1(X_1)+f_2(X_2)$, $Y=f_1(X_1)f_2(X_2)$. This disadvantage is compensated by adaptive boosting: in particular, additive interaction is not a problem at all, because the boosting is itself additive by construction. $\endgroup$
    – paperskilltrees
    Mar 31 at 18:25
  • $\begingroup$ @paperskilltress yeah I see what you are saying and you are correct that a simple tree will not give you those but I think in context what I said is fine. A tree will model interactions naturally as that is how the algorithm works and efficiently (really just meaning that it isn't exploding in parameters). This doesn't mean that it will necessarily approximate the true function or be a useful model of the universe just that it will take into account feature interactions without manual feature interactions / polynomials or a ton more parameters/nodes. $\endgroup$
    – Tylerr
    Apr 1 at 20:13
0
$\begingroup$

I would read elements of statistical learning to understand why gradient boosting does so well.

Forget about all this 'weak learner' which was what inspired it but is not really relevant.

Basically gradient boosting is an adaptive linear regression algorithm adding trees as input features. The combination of trees for capturing nonlinearity and linear methods for smoothing/reducing variance is statistically and computationally effective.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Do you know any particularly good packages implementing these combined tree/linear methods? $\endgroup$
    – Closed Limelike Curves
    Aug 25 at 1:36
  • $\begingroup$ Or something that might do better than trees for capturing nonlinearity? $\endgroup$
    – Closed Limelike Curves
    Aug 25 at 1:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.