I'm a bit perplexed after working on a kaggle data science competition for a month now. It's tabular data, all real/floating point numbers, on the order of maybe 100k examples, 100ish features, and a single output (real) number to predict. I've been using dense neural networks and have gotten good but middling results.
(edit - to give some context, most naive models get 30% error rate, trying hard with the NNs I get down to maybe 24-25% error rate, but then the best scores are all hitting 19-20%; and we have similar features.)
However consulting with people doing really well, they're all using things like XGBoost, LightGBM, etc. Looking into what these are, I see they are decision trees being optimized. I am confused why these work so well, or at least better than neural networks in certain contexts.
Can someone provide some context or background info for why these 'boosted decision tree' methods seem to be the state of the art for these types of data competitions? Would appreciate some insight, thanks!
Edit 2 - as a follow up/related question, could one 'boost' neural networks by fitting residuals from previous NNs? Do people do this/why don't they? I guess I'm just overall confused at trees being used so successfully for data regression, I don't understand how they could do so well vs. seemingly more numerically powerful and flexible algorithms.