# Neural networks: $tanh$ activativation function in output layer

In multi-class classification problem, if the output neurons are activated by $tanh$ function, how do we determine the class assignment for the pattern?

In case of sigmoid all output neurons would have value in $[0,1]$ and, using 1-of-C encoding of the response variable, the error is well-defined and the class determined by the neuron with maximum value. In case of using hyperbolic tangent, the neurons values would be in $[-1,1]$ and if we use 1-of-C encoding, the errors would be huge and it's not clear how do we classify the pattern based on the network's output?

Maybe, we should encode the response with $\{-1,+1\}$ with $+1$ for the correct class and $-1$ for the rest?

• If you are classifying into 1-of-C mutually exclusive classes, then you shouldn't be using tanh or logistic sigmoid output, you should be using a softmax and the log loss function. – alto Sep 20 '13 at 18:56
• alto, I do agree that these functions make more sense. However, all commercial packages offer you the choice for the output layer activation: identity, softmax, logistic, tanh... E.g. SPSS – Oleg Shirokikh Sep 20 '13 at 19:00
• And those would make since if you were doing something like multi-label classification, image denoising, etc. They do not if you are doing multi-class classification. – alto Sep 20 '13 at 21:11
• What do you mean by "multi-label" classification? What is the difference with "multi-class" classification? – Oleg Shirokikh Sep 20 '13 at 22:16

I believe one can rescale the output of tanh back into a probability simply using

$$\sigma(x) = \frac{1}{1+e^{-x}},$$

$$tanh(x) = 2\sigma(2x) - 1.$$

So to inverse transform the tanh output $a(x)$ back into a probability you do $0.5(a(x)+1)$.

• Could you explain what "$\sigma$" refers to--there's no appearance of it in the question--and elaborate on how an equation of two expressions in "$x$" can be used to "rescale" something? – whuber May 18 '16 at 17:45
• @whuber $\sigma$ is standard notation for the function on the rhs. It's called the sigmoid function. While it's popularity these days is due to it's use in neural nets, I believe it has a storied history in engineering. Because $\sigma(-\infty) = 0$ and $\sigma(\infty) = 1$, it is often used as an output function when one is modeling a probability. The second line is a mathematical identity between the sigmoid function and the hyperbolic tangent fn. Perhaps the choice of the word scaling is incorrect. – meh Jun 20 '16 at 2:17
• @ michek - I assume you mean- " you do " $0.5*(a(2x) + 1)$ ". I don't know (how to)/(if I'm allowed to) edit responses. – meh Jun 20 '16 at 2:21
• @ Oleg Since as Michek points out one can go by an almost linear transformation from tanh to sigmoid and back, there is nothing to be gained in using tanh as an activation as opposed to using the sigmoid as an activation function. I believe they both have the same issues with convergence in complicated situations. – meh Jun 20 '16 at 2:24
• @aginensky Thank you for clarifying the notation and the ideas behind it (+1). – whuber Jun 20 '16 at 13:14