How a softmax function can be used in multiclass classification?

Question

I'm trying to understand how a softmax function help classify in multi-class classification. In Andrew Ng video, he shows how a simple 1 layer neural network, with 2 inputs ($x_1, x_2$) and 3 outputs can be used to create a multi-class decision boundaries:

But, say I have these 3 linear decision boundaries: $$ x_2\\ x_2 - x_1 + 5\\x_2 + x_1 - 5 $$

And I'm looking at the point ($5,5$) which correspond to this:

This produces the same output, and when passed through the softmax will produce the same probability.

Now, I understand how you can tweak the weights of the 1-layer, so that this point will be classified one way or the other (e.g. multiply some decision boundary by some constant, say $y \rightarrow 5y$, and now we have a winner).

But it seems to me that, in the 1 layer architecture, all classifications will depend only on a single decision boundary at a time. And so you couldn't really create all these complex decision boundaries that Andrew shows, like:

These will require at least 2 layers, which will "inject" non-linearity into the decision boundaries.

Am I wrong, and if so, what am I missing?

Answer 1

So, after playing with this in GeoGebra, it turns out that it is possible only with 1 layer. My mistake seems to be that I thought that the lines in the examples correspond to the actual linear lines, or the Z's (i.e. the linear combinations of the inputs with the weights) - while in fact they are created from the "battle" of all Z's against each-other - e.g. if $z_1 = -x_1 + 2x_2 + 5$ and $z_2 = 2x_1 + 3x_2 -5$ - they will "battle" somewhere close to the line $-3x_1 - x_2 + 10$ (i.e. the sum of the two lines).

You can play with it yourself here.