1
$\begingroup$

Tree-based methods, e.g., random forests or GBRT use trees to construct most informative space partitioning. Given a single tree, in each partition, the value of the regression function (or a class) is computed using a simple model. For example, it can be a constant or a single label. Catboost's regression trees AFAIK can fit a linear regression.

In contrast, even simple 1-2 hidden layer neural networks (with non-linear activation) can beat GBRTs or random forests although it is not clear how a couple of matrix multiplications can apparently create region-specific regression or classification functions. Region-specific is a bit of a misnomer, but the point is that neural networks can apparently identify regions of interest and behave differently in different regions.

Is there a simple explanation of how this happens? Is there a complicated one (e.g., some theory)?

Many thanks!

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
0
$\begingroup$

In fact, the decision tree is equivalent with the deep neural network with the ReLU activation function.

See the paper Oblique Decision Trees from Derivatives of ReLU Networks or the or discussion on decision tree and neural networks.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you for the link. I've largely figured it out. Turns out there are number of papers that explain how these piecewise linear approximations work. In particular, you can always cut out a bounded polygon like shape by intersecting relu-defined planes. I put some of my thoughts in a blog (referencing some of the recent at that moment paper): searchivarius.org/blog/… $\endgroup$
    – Leonid Boytsov
    Commented Jun 27, 2020 at 22:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.