If your primary goal isn't to interpret values as probability values, but simply to find a "good" cost function for a tanh activation, in the sense that the neural network will learn faster the more "wrong" it is, then I believe that this may work:
or, in python form:
def C(y, a):
return -.5 * ( (1-y)*log(1-a) + (1+y)*log(1+a) ) + log(2)
More formally - this will give a cost function whose derivative is proportional to the error - ie, satisfy these two equations:
These are equations (71) and (72) from here: http://neuralnetworksanddeeplearning.com/chap3.html#the_cross-entropy_cost_function - thanks to Michael Nielsen for this great resource!
As others have noted, since it doesn't really have a probablity interpretation, I don't know if this formula can really be called a "cross-entropy" function for tanh... but it should be a "good" choice for a cost function for a tanh activation... or at least better than mean-squared-error!
As a caveat, though - I'm relatively new to all this, so feel free to point out if I've made any obvious errors!
EDIT: I made some code (using keras) to test the performance of this cost function, versus mean-squared-error, and my tests show nearly double the performance!
Here's the gist / code:
... and here were my results:
0.00850945741317 0.0121004378641 0.0144485289499 0.0259487312062 0.011714946112 0.0103978548285 0.0308227941615 0.0070723194485 0.0142580815076 0.0160464689688 0.013335006552 0.00783108495935 0.01220898181 0.0173612970037 0.0276861919335 0.0137742581066 0.0227281911188 0.0108267272463 0.0138346472817 0.0110096849345
maes: 0.00782039967299 0.0102608616831 0.00747869220265 0.0106438063825 0.00730774874144 0.0106295737387 0.00770924169349 0.014159071813 0.00763049735583 0.00902260985951 0.00602617238736 0.0103817401882 0.0108806326851 0.00879056832034 0.00743713011182 0.00648508079659 0.00660474073891 0.00766924505766 0.00738797156683 0.00708367751452
Not conclusive, obviously, but certainly suggestive that this will give better results than MSE for tanh activations!
EDIT 2: Realized the custom cost function was also causing the graph to evaluate slower, so # of iterations wasn't necessarily a fair comparison. Altered the test to compare error when running for the same amount of time, and then mse outperforms this tanh-cross-entropy-like cost. Still, it's possible it could be useful for more complicated graphs, when the additional time is a lower percentage of the total.