in order to solidify my understanding, I am doing some simple calculations with pen and paper for some very simple NN for binary classification (input vector with two entries, 1 hidden layer and just trying to comprehend every step of forward passing and backpropagation).
At the moment I'm stuck with one question: For binary classification I could go with one node in the output layer and use a sigmoid activation function or with two nodes in the output layer and use softmax. Mathematically, it isn't hard to show that sigmoid is the binary "special case" of the softmax and because of this, in other posts people often write that using softmax or sigmoid is the same for binary classification.
However, finding two equivalent NN implementations, one using one output and sigmoid, the other using two outputs and softmax, doesn't seem so trivial to me (I mean equivalent in the sense that I chose some weight initialization for each NN, feed them the same input and obtain the same prediction, without training). Surely the weights would have to be chosen in a specific way in order to obtain the exact same prediction from both NN. Would there be a way to initialize the weights so that we obtain the same prediction/to obtain the same predicted probability for class 1?
I'm not sure whether it is "stupid" to even search for two equivalent implementations without training, because after training both on the same training set, both should converge to some weights which will yield the same prediction. Or if I am missing something and it is indeed possible to easily go from one implementation to the other and construct the weights which would give me the same prediction?
Thanks in advance