I am using a tanh as my activation function for my NN. I also was using the cross entropy cost function previously when I had sigmoid neurons. The sigmoid neurons can never make it to zero but a tanh can and when I train the NN I will get division by zero errors. I switched back to the quadratic cost function but it converges slowly. Is there a way to use the cross entropy cost with a tanh or is there something better I could use?
Your question doesn't make any sense: a cross entropy loss means that you are making some distributional assumption.
One typical distributional assumption is that the target values are Bernoulli and the predictive distributions are Bernoulli. Therefore, the cross entropy is the expected surprisal. This is logistic regression. This is "logistic loss".
Another typical distributional assumption is normal with fixed variance. Then the cross entropy loss is equivalent to linear regression in the same way. This is the "quadratic loss".
Which distributional assumption corresponds to tanh?