# Can SVM and Decision Trees be seen as instances of neural networks?

We already know that neural networks with specific choices of activation function as well as connections can generalize large amount of ML models.

My question is: neural network also generalize SVM and Decision Trees?

It doesn't seem that NN does this so straight-forwardly.

• Might be of interest: "Neural Networks are Decision Trees" Commented Jul 19, 2023 at 9:11
• As far as I know The Universal Approximation Theorem holds for continuous functions. Decision Trees are not continuous. Commented Jul 19, 2023 at 11:36
• @ngmir just to make it clear to everyone else here, the "are" in that title is unidirectional: the article shows how to turn neural nets (with ReLU activations) into decision trees (in exactly the way you would expect), but not vice versa (which is not possible if we restrict our attention to neural nets which are continuous, which is necessary to train via gradient based methods). Commented Jul 21, 2023 at 19:44
• @Kozolovska - decision trees are universal approximators, just not the same theorem... and don't rely on the target function being differentiable, which if I recall correctly is part of the UAT for NNs (please correct me if I'm wrong about that!) Commented Jul 21, 2023 at 21:28
• I do not think this can be answered as stated in a meaningful way. What does it even mean "to generalize" something? What hypothesis spaces, finite or infinite sizes, which algos for hyperparameters and training etc. Why are you interested in this in the first place? Are you interested in this as a question of functional analysis, or would you like to understand practical problems?
– g g
Commented Jul 23, 2023 at 11:14

## Decision trees

You could write a decision tree as an artificial neural network (consisting of dense layers only). The activation function of the network would be $$\sigma(x)=\text{max}(0, \text{sign}(wx-b))$$ where bias $$b$$ is the value of the decision threshold in the corresponding tree node. Weight $$w\in[0,1]$$ activates or deactivates the connection.

## Support Vector Machine

Basic SVM is basically one neuron trained with a hinge loss.

$$\text{max}(0, 1-y_i(w^Tx_i-b))$$

When a kernel comes into play, you just use it as an activation function.

• "SVM is basically one neuron trained with a hinge loss." and a regularisation term. The regularisation term is critical to structural risk minimisation and how the SVM works. Commented Jul 28, 2023 at 18:46
• I can't understand how a decision tree can be written as a neural network. Based on your description, we can make an initial split (with the help of the activation function). But the next splits (second hidden layer) will see as features the activations from the first split (in contrast with a decision tree which will make its splits based on the original features). Commented Apr 22 at 15:07