Binary classification neural network - equivalent implementations with sigmoid and softmax

Question

in order to solidify my understanding, I am doing some simple calculations with pen and paper for some very simple NN for binary classification (input vector with two entries, 1 hidden layer and just trying to comprehend every step of forward passing and backpropagation).

At the moment I'm stuck with one question: For binary classification I could go with one node in the output layer and use a sigmoid activation function or with two nodes in the output layer and use softmax. Mathematically, it isn't hard to show that sigmoid is the binary "special case" of the softmax and because of this, in other posts people often write that using softmax or sigmoid is the same for binary classification.

However, finding two equivalent NN implementations, one using one output and sigmoid, the other using two outputs and softmax, doesn't seem so trivial to me (I mean equivalent in the sense that I chose some weight initialization for each NN, feed them the same input and obtain the same prediction, without training). Surely the weights would have to be chosen in a specific way in order to obtain the exact same prediction from both NN. Would there be a way to initialize the weights so that we obtain the same prediction/to obtain the same predicted probability for class 1?

I'm not sure whether it is "stupid" to even search for two equivalent implementations without training, because after training both on the same training set, both should converge to some weights which will yield the same prediction. Or if I am missing something and it is indeed possible to easily go from one implementation to the other and construct the weights which would give me the same prediction?

Thanks in advance

Answer 1

The pre-Softmax layer in a neural network produces some real 2-element vector $z_i$. If you choose $b_i = 0$ and $w_1 = 1$ and $w_2=0$, then $\text{Softmax}(z)_i = \sigma(z_i)$.

$$\begin{align} \text{Softmax}(x)_i &= \frac{\exp(x_i)}{\exp(x_1)+\exp(x_2)} \\ \text{Softmax}(z)_i &= \frac{\exp(w_i z_i + b_i)}{\exp(w_1 z_1 + b_1)+\exp(w_2 z_2 + b_2)} \\ \implies \sigma(z_1) &= \frac{\exp(z_1)}{\exp(z_1)+1} \\ \sigma(z_2) &= \frac{1}{\exp(z_1)+1} \\ \end{align}$$

This will work as long as the softmax and sigmoid networks have the same parameters (except for $w,b$). The easiest way to assure that is to just make a single model object that has options for sigmoid or softmax output.

TL;DR, multiply one of the inputs by 0.

See also: Sigmoid equivalent to Softmax exercise