2
$\begingroup$

I'm trying to understand how a softmax function help classify in multi-class classification. In Andrew Ng video, he shows how a simple 1 layer neural network, with 2 inputs ($x_1, x_2$) and 3 outputs can be used to create a multi-class decision boundaries:

enter image description here

But, say I have these 3 linear decision boundaries: $$ x_2\\ x_2 - x_1 + 5\\x_2 + x_1 - 5 $$

And I'm looking at the point ($5,5$) which correspond to this: enter image description here

This produces the same output, and when passed through the softmax will produce the same probability.

Now, I understand how you can tweak the weights of the 1-layer, so that this point will be classified one way or the other (e.g. multiply some decision boundary by some constant, say $y \rightarrow 5y$, and now we have a winner).

But it seems to me that, in the 1 layer architecture, all classifications will depend only on a single decision boundary at a time. And so you couldn't really create all these complex decision boundaries that Andrew shows, like:

enter image description here

These will require at least 2 layers, which will "inject" non-linearity into the decision boundaries.

Am I wrong, and if so, what am I missing?

$\endgroup$

1 Answer 1

1
$\begingroup$

So, after playing with this in GeoGebra, it turns out that it is possible only with 1 layer. My mistake seems to be that I thought that the lines in the examples correspond to the actual linear lines, or the Z's (i.e. the linear combinations of the inputs with the weights) - while in fact they are created from the "battle" of all Z's against each-other - e.g. if $z_1 = -x_1 + 2x_2 + 5$ and $z_2 = 2x_1 + 3x_2 -5$ - they will "battle" somewhere close to the line $-3x_1 - x_2 + 10$ (i.e. the sum of the two lines).

You can play with it yourself here.

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.