3
$\begingroup$

We already know that neural networks with specific choices of activation function as well as connections can generalize large amount of ML models.

My question is: neural network also generalize SVM and Decision Trees?

It doesn't seem that NN does this so straight-forwardly.

Please share your insights.

Improve this question
$\endgroup$
6
  • 6
    $\begingroup$ Might be of interest: "Neural Networks are Decision Trees" $\endgroup$
    – ngmir
    Commented Jul 19, 2023 at 9:11
  • $\begingroup$ As far as I know The Universal Approximation Theorem holds for continuous functions. Decision Trees are not continuous. $\endgroup$
    – Kozolovska
    Commented Jul 19, 2023 at 11:36
  • 2
    $\begingroup$ @ngmir just to make it clear to everyone else here, the "are" in that title is unidirectional: the article shows how to turn neural nets (with ReLU activations) into decision trees (in exactly the way you would expect), but not vice versa (which is not possible if we restrict our attention to neural nets which are continuous, which is necessary to train via gradient based methods). $\endgroup$
    – John Madden
    Commented Jul 21, 2023 at 19:44
  • 1
    $\begingroup$ @Kozolovska - decision trees are universal approximators, just not the same theorem... and don't rely on the target function being differentiable, which if I recall correctly is part of the UAT for NNs (please correct me if I'm wrong about that!) $\endgroup$
    – jbowman
    Commented Jul 21, 2023 at 21:28
  • 1
    $\begingroup$ I do not think this can be answered as stated in a meaningful way. What does it even mean "to generalize" something? What hypothesis spaces, finite or infinite sizes, which algos for hyperparameters and training etc. Why are you interested in this in the first place? Are you interested in this as a question of functional analysis, or would you like to understand practical problems? $\endgroup$
    – g g
    Commented Jul 23, 2023 at 11:14

1 Answer 1

Reset to default
7
+25
$\begingroup$

Decision trees

You could write a decision tree as an artificial neural network (consisting of dense layers only). The activation function of the network would be $\sigma(x)=\text{max}(0, \text{sign}(wx-b))$ where bias $b$ is the value of the decision threshold in the corresponding tree node. Weight $w\in[0,1]$ activates or deactivates the connection.

Support Vector Machine

Basic SVM is basically one neuron trained with a hinge loss.

$$\text{max}(0, 1-y_i(w^Tx_i-b))$$

When a kernel comes into play, you just use it as an activation function.

Improve this answer
$\endgroup$
2
  • 3
    $\begingroup$ "SVM is basically one neuron trained with a hinge loss." and a regularisation term. The regularisation term is critical to structural risk minimisation and how the SVM works. $\endgroup$
    – Dikran Marsupial
    Commented Jul 28, 2023 at 18:46
  • $\begingroup$ I can't understand how a decision tree can be written as a neural network. Based on your description, we can make an initial split (with the help of the activation function). But the next splits (second hidden layer) will see as features the activations from the first split (in contrast with a decision tree which will make its splits based on the original features). $\endgroup$
    – ado sar
    Commented Apr 22 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.