2
$\begingroup$

In the backpropagation algorithm when the output activation function is tanh and the number of classes is 2 (binary problem), the value obtained at the output layer is in the range between -1 to 1. The cross-entropy error function has log that is applied on the predicted values. Therefore, if one of the output values is a negative number, an invalid operation, namely, log (non-positive number), occurs, rendering the cross-entropy function invalid.

This boils down to the following questions:

  • Is it disallowed to set the output activation as tanh?

  • Should the output activation always be the softmax even for a binary class
    problem?

$\endgroup$
4
$\begingroup$

Typically, The softmax function is used in case of multi class problems and a single logistic function for binary classification. The reason is that the output nonlinearity and the loss "match", that means that the derivative is very simple--a property of generalized linear models.

On a side note, the tanh and the logistic sigmoid are related linearly. Tanh is just the logistic scaled and translated from the $[0, 1]$ to the $[-1, 1]$ interval.

$\endgroup$
  • $\begingroup$ You are right, however, like I said, the cross-entropy cost function, in python syntax, is cost = -np.sum(Y * np.log(a_output)) (consider the log) where Y is the real output and a_output constitutes the predicted values. Since tanh can yield negative values, then np.log(a_output) can produce a math domain error. This is what is confusing me, the fact that there is a possibility that log is applied on negative predicted values, causing a math error. $\endgroup$ – Curious Jul 30 '13 at 7:24
  • 3
    $\begingroup$ As bayerj suggests (+1), the tanh is just a rescaling of the logistic sigmoid (which is the normal choice for binary regression). Just replace a_output with 0.5*(a_output + 1) and this will undo the scaling. However, there is no good reason to use tanh for binary classification, just use the logistic function instead. See Bishop's book on Neural Networks for Pattern Recognition for a good discussion of the relationship between cost functions and output layer activation function (and neural nets in general). $\endgroup$ – Dikran Marsupial Jul 30 '13 at 9:58
  • $\begingroup$ Sorry for not being clear enough on this. Thanks to Dikran for compensating that. $\endgroup$ – bayerj Jul 30 '13 at 10:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.