1
$\begingroup$

We have these two mappings:

1)

000→0
001→0
010→0 
011→1 
100→0 
101→1 
110→1
111→1

2)

000→0
001→1
010→1 
011→0 
100→1 
101→0 
110→0
111→0

This is supposed to be able to learn 1 but not 2. I was thinking of just drawing this dawn and add 3 fake weights for each input and see how it does. Thing is, if there is some intuitive way to understand why 1 can be learned while the 2nd not.

cheers!

$\endgroup$

1 Answer 1

1
$\begingroup$

The fake weights have made the problem linearly non-separabile, which means there does not exist a plane in the 3D space that can separate the points perfectly (with zeros on one side and ones on the other side).

Something like this, we can't find a plane that separates the points exactly. But if we change the origin (000) to be 1, it will become linearly separable.

something like this

It is a 3D analogy to the XOR problem: http://www.ece.utep.edu/research/webfuzzy/docs/kk-thesis/kk-thesis-html/node19.html

A single layer perceptron/network is essentially a generalized linear model, which means it can only learn a linear decision boundary, so it will fail the second case.

For instance, a single layer network can be represented as $f(wx+b)$. In the first case if we simply let $f(wx+b)$ be $x_1+x_2+x_3-1.5$, then points with label 1 will have values greater than zero, and points with label 0 will have values smaller than zero. So it is linearly separable.

In the second case we can't find such $w$ and $b$ that can separate the data perfectly, so the single layer network will fail in this case. There're many ways to solve such problem, like constructing new features, or using more hidden layers.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.