Tree-based methods, e.g., random forests or GBRT use trees to construct most informative space partitioning. Given a single tree, in each partition, the value of the regression function (or a class) is computed using a simple model. For example, it can be a constant or a single label. Catboost's regression trees AFAIK can fit a linear regression.

In contrast, even simple 1-2 hidden layer neural networks (with non-linear activation) can beat GBRTs or random forests although it is not clear how a couple of matrix multiplications can apparently create region-specific regression or classification functions. Region-specific is a bit of a misnomer, but the point is that neural networks can apparently identify regions of interest and behave differently in different regions.

Is there a simple explanation of how this happens? Is there a complicated one (e.g., some theory)?

Many thanks!

Your Answer


By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.