0
$\begingroup$

I am using a tanh as my activation function for my NN. I also was using the cross entropy cost function previously when I had sigmoid neurons. The sigmoid neurons can never make it to zero but a tanh can and when I train the NN I will get division by zero errors. I switched back to the quadratic cost function but it converges slowly. Is there a way to use the cross entropy cost with a tanh or is there something better I could use?

$\endgroup$
  • $\begingroup$ When exactly do you get divizion by zero errors? $\endgroup$ – user31264 May 5 '16 at 15:29
  • $\begingroup$ When any of the outputs of the neural net are zero. $\endgroup$ – chasep255 May 5 '16 at 15:38
  • 1
    $\begingroup$ tanh returns values from -1 to 1. Cross-entropy assumes that values are from 0 to 1. How do you use cross entropy with tanh? $\endgroup$ – user31264 May 5 '16 at 16:13
  • $\begingroup$ Ok. Is there a better cost function to use for tanh $\endgroup$ – chasep255 May 5 '16 at 16:18
  • $\begingroup$ Am I correct that your outputs of the neural net are from -1 to 1? (many of them are negative)? $\endgroup$ – user31264 May 5 '16 at 16:21
1
$\begingroup$

It's common to use softmax as a final layer. It helps you to convert the output values to the probabilities. If you use softmax as an activation function for the final layer you can use any function you like for the previous layers.

$\endgroup$
1
$\begingroup$

Your question doesn't make any sense: a cross entropy loss means that you are making some distributional assumption.

One typical distributional assumption is that the target values are Bernoulli and the predictive distributions are Bernoulli. Therefore, the cross entropy is the expected surprisal. This is logistic regression. This is "logistic loss".

Another typical distributional assumption is normal with fixed variance. Then the cross entropy loss is equivalent to linear regression in the same way. This is the "quadratic loss".

Which distributional assumption corresponds to tanh?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.