In the backpropagation algorithm when the output activation function is
tanh and the number of classes is 2 (binary problem), the value obtained at the output layer is in the range between -1 to 1. The cross-entropy error function has
log that is applied on the predicted values. Therefore, if one of the output values is a negative number, an invalid operation, namely,
log (non-positive number), occurs, rendering the cross-entropy function invalid.
This boils down to the following questions:
Is it disallowed to set the output activation as
Should the output activation always be the softmax even for a binary class