# How to use 1.7159 * tanh(2/3 * x) as activation function?

I have a simple neural network and it works with the logistic function as activation function. Now I want to avoid the saturation problem by substituting the logistic function by the hyperbolic tangent:

#define SIGMOID(x) (1.7159*tanh(0.66666667*x))
#define DSIGMOID(S) (0.666666667/1.7159*(1.7159-(S))*(1.7159+(S)))


But the network never converges, the MSE stays the same throughout the training. Here's my training samples:

double training_data[][4]={
{0, 0,  0,  -1},
{0, 0,  1,  1},
{0, 1,  0,  1},
{0, 1,  1,  -1},
{1, 0,  0,  1},
{1, 0,  1,  -1},
{1, 1,  0,  -1},
{1, 1,  1,  1}};


The network does converge if I use the original (non-scaled) hyperbolic tangent function, that is:

#define SIGMOID(x) (tanh(x))
#define DSIGMOID(S) (1-((S)*(S)))


Do I miss something? E.g. Scaling the output to match the range (-1.7159, 1.7159) or anything?

• I am unfamiliar with your programming language, is it C? In MatLab many functions that work with or respond to built-in (libraried) functions need to be expressed in vector notation. Instead of using the scalar multiplication "", it is more desirable to use the vectorized "." operator. This improves execution speed substantially. May 27, 2013 at 23:29
• 2/3 will give a better approximation than 0.666666667 in most languages.... Presumably of little consequence, but best to do it properly. May 27, 2013 at 23:59
• @EngrStudent This is C, no Matlab
– Max
May 28, 2013 at 0:50
• @NickCox There's no real difference between 0.666666667 and 2/3. But using 2/3 is more expensive because the computer will have perform additional calculation. Most people will just hardcode it that way.
– Max
May 28, 2013 at 0:50
• The first example I Googled used a different approximation. More importantly, I suspect all you need to do is to calculate the best approximation to 2/3 just once and define it as a real constant if the expense worries you. The details will depend on your language. May 28, 2013 at 0:54

When I plot using the following R-code:

x <- seq(from = -2, to = 2, by = 0.01 )
y <- (0.666666667/1.7159*(1.7159-(x))*(1.7159+(x)))
y2 <- (1.7159*tanh(0.66666667*x))

plot(x,y2,col = "red")
points(x,y)


I get the following plot:

One of these is a sigmoid (red), one is not a great derivative (black). Notice the negative values. This is going to define a radius of convergence that shoots Newtons-methods toward infinity.

Now using this R-code:

x <- seq(from = -2, to = 2, by = 0.01 )
y <- 1.14393*(1/cosh(2*x/3))^2
y2 <- (1.7159*tanh(0.66666667*x))

plot(x,y2,col = "red", type = "b")
points(x,y)


I get this plot:

It is a more plausible graph of the derivative(black) for the sigmoid(red).

Edit:

Here are some basics on Tanh and friends.

Please notice in link 1 that the derivative of Hyperbolic Tangent is pow( hyperbolic_secant,2) and not pow( hyperbolic_cosine,2).

• I don't know what a hyperbolic secant is, but the derivative of f(x) = tanh(x) is f'(x) = 1-pow(tanh(x), 2). The formula I use is absolutely correct (because I just copied and paste it from a open source library), so it seems the hyperbolic secant is wrong. I think the problem with my NN has something to do with the output (e.g. remap it from (-1.7159, 1.7159) to (-1, 1)... but I haven't seen anyone do that)
– Max
May 28, 2013 at 7:00
• "absolutely correct" because copied and pasted "from a open source library": would that it were so.... May 28, 2013 at 14:34
• @NickCox This page tells a different story. Derivative of tanh(x) is 1-pow(tanh(x), 2): math2.org/math/derivatives/more/hyperbolics.htm. And the library I copied and pasted from is the famous FANN, so it must be absolutely correct. The original tanh(x) works as a proof that I got the derivative correctly
– Max
May 29, 2013 at 23:52
• Hehe. When there are two mutually exclusive and also true statements then there must be a third statement that resolves them. If $D = 1-tanh^2(x)$ and $D = sech^2(x)$ then $tanh^2(x) + sech^2(x) = 1$. It is about 1/3 the way down this page (en.wikipedia.org/wiki/Hyperbolic_function). Neither of these are the quadratic in the first DSIGMOID. "(0.666666667/1.7159*(1.7159-(S))*(1.7159+(S)))" Why is that statement correct? Within a radius of about 2-miles the flat-world hypothesis is a good approximation to the sphere of the earth. Counterexamples disprove, examples don't prove. May 30, 2013 at 1:48
• Not a good idea to be dogmatic when you don't know key facts. May 30, 2013 at 6:29

FANN_SIGMOID_SYMMETRIC

f2M_set_act_function_hidden (ann, FANN_SIGMOID_SYMMETRIC);

f2M_set_act_function_output (ann, FANN_SIGMOID_SYMMETRIC);

Symmetric sigmoid activation function, AKA tanh. One of the most used activation functions.

This activation function gives output that is between -1 and 1.