I know the cross entropy function can be used as the cost function, if the activation function is logistic function: i.e.: $\frac{1}{1 + e^{-x}}$

However, I just wonder:

Can the cross entropy cost function be used with many other activation functions, such as tanh?


2 Answers 2


Yes we can, as long as we use some normalizor (e.g. softmax) to ensure that the final output values are in between 0 and 1 and add up to 1.

If you're doing binary classification and only use one output value, only normalizing it to be between 0 and 1 will do.

As mentioned by Sycorax, depending on what procedure you use to shifting and rescaling tanh back to [0,1], you could end up with a logistic unit again, see this answer.

  • $\begingroup$ tanh output between -1 and +1, so can it not be used with cross entropy cost function? $\endgroup$
    – xmllmx
    Jul 3, 2016 at 11:22
  • $\begingroup$ @xmllmx not really, cross entropy requires the output can be interpreted as probability values, so we need some normalization for that. $\endgroup$
    – dontloo
    Jul 3, 2016 at 11:26
  • 1
    $\begingroup$ Depending on what procedure you use to shifting and rescaling $\tanh$ back to $[0,1]$, you could end up with a logistic unit again -- since the logistic unit is just a shifted-rescaled $\tanh$ unit. $\endgroup$
    – Sycorax
    Jul 7, 2018 at 22:41

If your primary goal isn't to interpret values as probability values, but simply to find a "good" cost function for a tanh activation, in the sense that the neural network will learn faster the more "wrong" it is, then I believe that this may work:


or, in python form:

def C(y, a):
    return  -.5 * ( (1-y)*log(1-a) + (1+y)*log(1+a) ) + log(2)

More formally - this will give a cost function whose derivative is proportional to the error - ie, satisfy these two equations:



These are equations (71) and (72) from here: http://neuralnetworksanddeeplearning.com/chap3.html#the_cross-entropy_cost_function - thanks to Michael Nielsen for this great resource!

As others have noted, since it doesn't really have a probablity interpretation, I don't know if this formula can really be called a "cross-entropy" function for tanh... but it should be a "good" choice for a cost function for a tanh activation... or at least better than mean-squared-error!

As a caveat, though - I'm relatively new to all this, so feel free to point out if I've made any obvious errors!

EDIT: I made some code (using keras) to test the performance of this cost function, versus mean-squared-error, and my tests show nearly double the performance!

Here's the gist / code: https://gist.github.com/elrond79/a016230229322f3ba2b46a99ff0b4e7c

... and here were my results:

0.00850945741317 0.0121004378641 0.0144485289499 0.0259487312062 0.011714946112 0.0103978548285 0.0308227941615 0.0070723194485 0.0142580815076 0.0160464689688 0.013335006552 0.00783108495935 0.01220898181 0.0173612970037 0.0276861919335 0.0137742581066 0.0227281911188 0.0108267272463 0.0138346472817 0.0110096849345

average: 0.01509578457033614

maes: 0.00782039967299 0.0102608616831 0.00747869220265 0.0106438063825 0.00730774874144 0.0106295737387 0.00770924169349 0.014159071813 0.00763049735583 0.00902260985951 0.00602617238736 0.0103817401882 0.0108806326851 0.00879056832034 0.00743713011182 0.00648508079659 0.00660474073891 0.00766924505766 0.00738797156683 0.00708367751452

average: 0.008570473125530578

Not conclusive, obviously, but certainly suggestive that this will give better results than MSE for tanh activations!

EDIT 2: Realized the custom cost function was also causing the graph to evaluate slower, so # of iterations wasn't necessarily a fair comparison. Altered the test to compare error when running for the same amount of time, and then mse outperforms this tanh-cross-entropy-like cost. Still, it's possible it could be useful for more complicated graphs, when the additional time is a lower percentage of the total.

  • $\begingroup$ In my opinion, if the dC/dz contains other terms other than (a-y), it usually makes the gradient to be biased based on the a value, which is unwanted side-effect. Thus there is an optimal C which is integrate (a-y)/g'(z(a)). In this case it will be 1/2 ((-1 + y) log(1 - x) - (1 + y) log(1 + x)). Your edit2 seems strange. Because evaluation does not require to compute C at all and in real inference application we always turn off the loss evaluation. $\endgroup$
    – Wang
    Jan 23, 2023 at 13:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.