Why is XGBoost so Good? And Boosting/Trees in General? I'm a bit perplexed after working on a kaggle data science competition for a month now. It's tabular data, all real/floating point numbers, on the order of maybe 100k examples, 100ish features, and a single output (real) number to predict. I've been using dense neural networks and have gotten good but middling results.
(edit - to give some context, most naive models get 30% error rate, trying hard with the NNs I get down to maybe 24-25% error rate, but then the best scores are all hitting 19-20%; and we have similar features.)
However consulting with people doing really well, they're all using things like XGBoost, LightGBM, etc. Looking into what these are, I see they are decision trees being optimized. I am confused why these work so well, or at least better than neural networks in certain contexts.
Can someone provide some context or background info for why these 'boosted decision tree' methods seem to be the state of the art for these types of data competitions? Would appreciate some insight, thanks!
Edit 2 - as a follow up/related question, could one 'boost' neural networks by fitting residuals from previous NNs? Do people do this/why don't they? I guess I'm just overall confused at trees being used so successfully for data regression, I don't understand how they could do so well vs. seemingly more numerically powerful and flexible algorithms.
 A: Here is my hand-wavey attempt for why those models perform unusually well in Kaggle and other data science competitions.
Trees can handle a ton of variables (even highly correlated ones) quite well because of the procedure in which they access them.  A (CART) Tree won't use all variables to fit the next split but instead pick the next best one to create the decision.  This allows us to throw everything in a data science competition at a tree-based model and quickly see what sticks.  This is unlike a linear regression which will fit all parameters simultaneously so special care is needed when using a lot of variables.
Trees will model feature interactions naturally and efficiently.  This makes trees really powerful in competitions because we usually have no domain knowledge or the variables themselves have been encoded. For linear regression we usually have to add dimensions via interaction terms or polynomial expansions in order to inject that knowledge but a tree with a depth of more than one is already doing it.  Similarly for neural nets we have to add more layers which increases the parameters bigly.
So, trees are great but why not a random forest or some other ensembling technique rather than boosting? No idea, but boosting is great and specifically xgboost and lightgbm implementations use regularized trees along with algos which help speed up computation (like histogram splits) to deal with boosting's biggest computational issue: it is sequentially learned. And a lot of their development have been influenced greatly by kaggle and other data science competitions.
For your edit yes you could boost any algorithm since it is just an ensembling procedure but you do not get similar gains with algorithms like a NN or linear regression as you would with trees.  This is because adding two trees adds complexity where as adding two sets of coefficients in linear regression leaves you with the same linear regression formula.  Neural nets also can overfit so it is not a good candidate for boosting but there are are still relevant algos such as the resnet architecture which is theoretically related to boosting, see here.
A: I would read elements of statistical learning to understand why gradient boosting does so well.
Forget about all this 'weak learner' which was what inspired it but is not really relevant.
Basically gradient boosting is an adaptive linear regression algorithm adding trees as input features. The combination of trees for capturing nonlinearity and  linear methods for smoothing/reducing variance is statistically and computationally effective.
