I'm trying to understand how a softmax function help classify in multi-class classification. In Andrew Ng video, he shows how a simple 1 layer neural network, with 2 inputs ($x_1, x_2$) and 3 outputs can be used to create a multi-class decision boundaries:
But, say I have these 3 linear decision boundaries: $$ x_2\\ x_2 - x_1 + 5\\x_2 + x_1 - 5 $$
And I'm looking at the point ($5,5$) which correspond to this:
This produces the same output, and when passed through the softmax will produce the same probability.
Now, I understand how you can tweak the weights of the 1-layer, so that this point will be classified one way or the other (e.g. multiply some decision boundary by some constant, say $y \rightarrow 5y$, and now we have a winner).
But it seems to me that, in the 1 layer architecture, all classifications will depend only on a single decision boundary at a time. And so you couldn't really create all these complex decision boundaries that Andrew shows, like:
These will require at least 2 layers, which will "inject" non-linearity into the decision boundaries.
Am I wrong, and if so, what am I missing?