I know the cross entropy function can be used as the cost function, if the activation function is logistic function: i.e.: $\frac{1}{1 + e^{-x}}$

However, I just wonder:

Can the cross entropy cost function be used with many other activation functions, such as tanh?


Yes we can, as long as we use some normalizor (e.g. softmax) to ensure that the final output values are in between 0 and 1 and add up to 1.

If you're doing binary classification and only use one output value, only normalizing it to be between 0 and 1 will do.

As mentioned by Sycorax, depending on what procedure you use to shifting and rescaling tanh back to [0,1], you could end up with a logistic unit again, see this answer.

| cite | improve this answer | |
  • $\begingroup$ tanh output between -1 and +1, so can it not be used with cross entropy cost function? $\endgroup$ – xmllmx Jul 3 '16 at 11:22
  • $\begingroup$ @xmllmx not really, cross entropy requires the output can be interpreted as probability values, so we need some normalization for that. $\endgroup$ – dontloo Jul 3 '16 at 11:26
  • 1
    $\begingroup$ Depending on what procedure you use to shifting and rescaling $\tanh$ back to $[0,1]$, you could end up with a logistic unit again -- since the logistic unit is just a shifted-rescaled $\tanh$ unit. $\endgroup$ – Sycorax Jul 7 '18 at 22:41

If your primary goal isn't to interpret values as probability values, but simply to find a "good" cost function for a tanh activation, in the sense that the neural network will learn faster the more "wrong" it is, then I believe that this may work:


or, in python form:

def C(y, a):
    return  -.5 * ( (1-y)*log(1-a) + (1+y)*log(1+a) ) + log(2)

More formally - this will give a cost function whose derivative is proportional to the error - ie, satisfy these two equations:



These are equations (71) and (72) from here: http://neuralnetworksanddeeplearning.com/chap3.html#the_cross-entropy_cost_function - thanks to Michael Nielsen for this great resource!

As others have noted, since it doesn't really have a probablity interpretation, I don't know if this formula can really be called a "cross-entropy" function for tanh... but it should be a "good" choice for a cost function for a tanh activation... or at least better than mean-squared-error!

As a caveat, though - I'm relatively new to all this, so feel free to point out if I've made any obvious errors!

EDIT: I made some code (using keras) to test the performance of this cost function, versus mean-squared-error, and my tests show nearly double the performance!

Here's the gist / code: https://gist.github.com/elrond79/a016230229322f3ba2b46a99ff0b4e7c

... and here were my results:

0.00850945741317 0.0121004378641 0.0144485289499 0.0259487312062 0.011714946112 0.0103978548285 0.0308227941615 0.0070723194485 0.0142580815076 0.0160464689688 0.013335006552 0.00783108495935 0.01220898181 0.0173612970037 0.0276861919335 0.0137742581066 0.0227281911188 0.0108267272463 0.0138346472817 0.0110096849345

average: 0.01509578457033614

maes: 0.00782039967299 0.0102608616831 0.00747869220265 0.0106438063825 0.00730774874144 0.0106295737387 0.00770924169349 0.014159071813 0.00763049735583 0.00902260985951 0.00602617238736 0.0103817401882 0.0108806326851 0.00879056832034 0.00743713011182 0.00648508079659 0.00660474073891 0.00766924505766 0.00738797156683 0.00708367751452

average: 0.008570473125530578

Not conclusive, obviously, but certainly suggestive that this will give better results than MSE for tanh activations!

EDIT 2: Realized the custom cost function was also causing the graph to evaluate slower, so # of iterations wasn't necessarily a fair comparison. Altered the test to compare error when running for the same amount of time, and then mse outperforms this tanh-cross-entropy-like cost. Still, it's possible it could be useful for more complicated graphs, when the additional time is a lower percentage of the total.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.