1
$\begingroup$

Support Vector machines employ the kernel trick in order to find a space where the data is mostly linearly separable and then determine what the appropriate hyperplane. However, back in the original space, the hyperplane would look non-linear.

Neural networks also learn a non-linear decision boundary through the use of the non-linear activation functions. However, the output representation of the final layer is therefore the representation that will be used to finally classify the data.

In the case of a binary classification task, it the case that the best representation that can be learned is likely to one that allows data to be linearly separable?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
0
$\begingroup$

If you use non-linear activation functions (e.g. ReLu, tanh, logit) and have at least one hidden layer, then the neural-network can indeed learn non-linear separation boundaries.

Solving the XOR function is a famous historical example of using a NN with a hidden layer to classify data that are not linearly separable. This article shows it nicely and I stole the image from the bottom of the article illustrating the decision boundary learned for data that are not linearly separable.

enter image description here

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for your response. Yes it is a non linear decision boundary but my question is more about whether the neural network is “lifting the space” in the way the kernel trick does to separate the data. I believe that an appropriate kernel can also solve XOR. $\endgroup$
    – tryingtolearn
    Commented Dec 4, 2018 at 23:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.