Implicit space partitiong in neural networks vs space partitioning in tree-based methods

Question

Tree-based methods, e.g., random forests or GBRT use trees to construct most informative space partitioning. Given a single tree, in each partition, the value of the regression function (or a class) is computed using a simple model. For example, it can be a constant or a single label. Catboost's regression trees AFAIK can fit a linear regression.

In contrast, even simple 1-2 hidden layer neural networks (with non-linear activation) can beat GBRTs or random forests although it is not clear how a couple of matrix multiplications can apparently create region-specific regression or classification functions. Region-specific is a bit of a misnomer, but the point is that neural networks can apparently identify regions of interest and behave differently in different regions.

Is there a simple explanation of how this happens? Is there a complicated one (e.g., some theory)?

Many thanks!

Answer 1

In fact, the decision tree is equivalent with the deep neural network with the ReLU activation function.

See the paper Oblique Decision Trees from Derivatives of ReLU Networks or the or discussion on decision tree and neural networks.