# Can I use tanh activation function in the output layer for binary classification?

I am new in the field of machine learning. So this question may sound silly. We usually use $sigmoid$ in output layer for binary classification. In my experiments, I found that $tanh$ gives higher accuracy and lower binary cross entropy loss if $sigmoid$ is replaced with $tanh$ in the output layer. Can someone please explain the possible reason? I am using labels as $0$ and $1$.

The code is shown below. I am using Keras with TensorFlow in the backend.

input_shape = (200, )
left_input = Input(input_shape)
right_input = Input(input_shape)

model = Sequential()

x1 = model(left_input)
x2 = model(right_input)

dotted = Dot(axes=1,normalize=True)([x1, x2])
out = Dense(1,activation='sigmoid',kernel_initializer='glorot_uniform',bias_initializer='zeros')(dotted)

siamese = Model(inputs=[left_input, right_input], outputs=out)

• Is by any chance your dataset biased towards the 0 class? – Lugi Feb 21 '18 at 14:08
• Thanks Lugi for your comment. No. In my experiments, both classes have an equal number of samples. – talk2speech Feb 21 '18 at 14:24
• Hyperbolic tangent is just a rescaling of the logistic function. $$f(x) = \frac{\exp(x)}{1 + \exp(x)} \\ \tanh(x) = 2f(2x) - 1$$ However, $\tanh(x)\in[-1,1]$, so it's not clear how you're computing cross-entropy loss. Cross entropy loss is using logarithms of probabilities, and logarithms of negative numbers are not real. Are you sure this isn't just an artifact of clipping $\tanh(x)$ when it's non-positive? – Sycorax says Reinstate Monica Feb 21 '18 at 14:44
• Is your question a duplicate of this one? stats.stackexchange.com/questions/221901/… – Sycorax says Reinstate Monica Feb 22 '18 at 1:22
• Thank you very much, Sycorax for the explanation and the pointer another related question. I am computing binary cross entropy in the same method as you explained. But it is not still clear to me why I am getting higher performance (in terms of accuracy for both training and test set) when I use tanh instead of standard sigmoid. I checked the output; they are always between 0 and 1. – talk2speech Feb 22 '18 at 12:39

The line dotted = Dot(axes=1,normalize=True)([x1, x2]) computes the cosine of the angle $\theta$ between x1 and x2. If it's always true that $\cos(\theta)>0$, that implies $0 < \tanh(\cos(\theta)) < 1$. Under these conditions, this resolves the riddle of how you're getting proper probabilities using $\tanh$. But remember that you're applying a linear transformation, rather than $\tanh(\cos(\theta))$ directly, so you further require that even after applying the linear transformation, the bounds are still respected.
As for why performance for $\tanh$ is better than $\text{sigmoid}$ in this case, it could be the usual reason that NN researcher suggest: $\tanh$ has steeper gradients, so backprop is more effective.
• Exactly! Thanks a lot for your help in understanding the problem. Thanks Lugi for asking me to share the code. In most cases, they are greater than $0$. Could you please suggest some references "$tanh$ has steeper gradients, so backprop is more effective." This will help to understand the problem in details and related background. – talk2speech Feb 22 '18 at 16:26