I hope this question helps shed some light on trees vs neural models. I recently came across a model tree, or a recursive partitioning model. It is basically a decision tree that has linear regression models trained for every leaf node of this tree. A most prominent example of this method could be the M5 model from Quinlan, 1992. Basically what got me here is this thread. I thought about this model and it seemed to me that it is relatively similar to a feed-forward neural network using ReLU as activation functions. I played around a little bit with this nice visualization tool from Karpathy. Try playing around with how only ReLU activations would get joined together to create a nice approximation for the input data.
I was wondering if anyone could explain the main, fundamental differences about what these models can or cannot do; or in what way they are the same/different. All your answers are appreciated!