# Division by zero with cross entropy cost function

I am using a tanh as my activation function for my NN. I also was using the cross entropy cost function previously when I had sigmoid neurons. The sigmoid neurons can never make it to zero but a tanh can and when I train the NN I will get division by zero errors. I switched back to the quadratic cost function but it converges slowly. Is there a way to use the cross entropy cost with a tanh or is there something better I could use?

• When exactly do you get divizion by zero errors? May 5 '16 at 15:29
• When any of the outputs of the neural net are zero. May 5 '16 at 15:38
• tanh returns values from -1 to 1. Cross-entropy assumes that values are from 0 to 1. How do you use cross entropy with tanh? May 5 '16 at 16:13
• Ok. Is there a better cost function to use for tanh May 5 '16 at 16:18
• Am I correct that your outputs of the neural net are from -1 to 1? (many of them are negative)? May 5 '16 at 16:21

## 2 Answers

It's common to use softmax as a final layer. It helps you to convert the output values to the probabilities. If you use softmax as an activation function for the final layer you can use any function you like for the previous layers.

Your question doesn't make any sense: a cross entropy loss means that you are making some distributional assumption.

One typical distributional assumption is that the target values are Bernoulli and the predictive distributions are Bernoulli. Therefore, the cross entropy is the expected surprisal. This is logistic regression. This is "logistic loss".

Another typical distributional assumption is normal with fixed variance. Then the cross entropy loss is equivalent to linear regression in the same way. This is the "quadratic loss".

Which distributional assumption corresponds to tanh?