1
$\begingroup$

I have a simple neural network and it works with the logistic function as activation function. Now I want to avoid the saturation problem by substituting the logistic function by the hyperbolic tangent:

#define SIGMOID(x) (1.7159*tanh(0.66666667*x)) 
#define DSIGMOID(S) (0.666666667/1.7159*(1.7159-(S))*(1.7159+(S)))

But the network never converges, the MSE stays the same throughout the training. Here's my training samples:

double training_data[][4]={
            {0, 0,  0,  -1},
            {0, 0,  1,  1},
            {0, 1,  0,  1},
            {0, 1,  1,  -1},
            {1, 0,  0,  1},
            {1, 0,  1,  -1},
            {1, 1,  0,  -1},
            {1, 1,  1,  1}};

The network does converge if I use the original (non-scaled) hyperbolic tangent function, that is:

#define SIGMOID(x) (tanh(x))
#define DSIGMOID(S) (1-((S)*(S)))

Do I miss something? E.g. Scaling the output to match the range (-1.7159, 1.7159) or anything?

Improve this question
$\endgroup$
11
  • $\begingroup$ I am unfamiliar with your programming language, is it C? In MatLab many functions that work with or respond to built-in (libraried) functions need to be expressed in vector notation. Instead of using the scalar multiplication "", it is more desirable to use the vectorized "." operator. This improves execution speed substantially. $\endgroup$
    – EngrStudent
    May 27, 2013 at 23:29
  • $\begingroup$ 2/3 will give a better approximation than 0.666666667 in most languages.... Presumably of little consequence, but best to do it properly. $\endgroup$
    – Nick Cox
    May 27, 2013 at 23:59
  • $\begingroup$ @EngrStudent This is C, no Matlab $\endgroup$
    – Max
    May 28, 2013 at 0:50
  • 1
    $\begingroup$ @NickCox There's no real difference between 0.666666667 and 2/3. But using 2/3 is more expensive because the computer will have perform additional calculation. Most people will just hardcode it that way. $\endgroup$
    – Max
    May 28, 2013 at 0:50
  • $\begingroup$ The first example I Googled used a different approximation. More importantly, I suspect all you need to do is to calculate the best approximation to 2/3 just once and define it as a real constant if the expense worries you. The details will depend on your language. $\endgroup$
    – Nick Cox
    May 28, 2013 at 0:54

2 Answers 2

Reset to default
3
$\begingroup$

When I plot using the following R-code:

x <- seq(from = -2, to = 2, by = 0.01 )
y <- (0.666666667/1.7159*(1.7159-(x))*(1.7159+(x)))
y2 <- (1.7159*tanh(0.66666667*x)) 

plot(x,y2,col = "red")
points(x,y)

I get the following plot: plot of the give expressions

One of these is a sigmoid (red), one is not a great derivative (black). Notice the negative values. This is going to define a radius of convergence that shoots Newtons-methods toward infinity.

Now using this R-code:

x <- seq(from = -2, to = 2, by = 0.01 )
y <- 1.14393*(1/cosh(2*x/3))^2
y2 <- (1.7159*tanh(0.66666667*x)) 

plot(x,y2,col = "red", type = "b")
points(x,y)

I get this plot: updated plot of expressions

It is a more plausible graph of the derivative(black) for the sigmoid(red).

This was fun: link.

Edit:

Here are some basics on Tanh and friends.

  1. http://mathworld.wolfram.com/HyperbolicTangent.html
  2. http://mathworld.wolfram.com/HyperbolicCosine.html
  3. http://mathworld.wolfram.com/HyperbolicSine.html

Please notice in link 1 that the derivative of Hyperbolic Tangent is pow( hyperbolic_secant,2) and not pow( hyperbolic_cosine,2).

Improve this answer
$\endgroup$
7
  • $\begingroup$ I don't know what a hyperbolic secant is, but the derivative of f(x) = tanh(x) is f'(x) = 1-pow(tanh(x), 2). The formula I use is absolutely correct (because I just copied and paste it from a open source library), so it seems the hyperbolic secant is wrong. I think the problem with my NN has something to do with the output (e.g. remap it from (-1.7159, 1.7159) to (-1, 1)... but I haven't seen anyone do that) $\endgroup$
    – Max
    May 28, 2013 at 7:00
  • 1
    $\begingroup$ "absolutely correct" because copied and pasted "from a open source library": would that it were so.... $\endgroup$
    – Nick Cox
    May 28, 2013 at 14:34
  • $\begingroup$ @NickCox This page tells a different story. Derivative of tanh(x) is 1-pow(tanh(x), 2): math2.org/math/derivatives/more/hyperbolics.htm. And the library I copied and pasted from is the famous FANN, so it must be absolutely correct. The original tanh(x) works as a proof that I got the derivative correctly $\endgroup$
    – Max
    May 29, 2013 at 23:52
  • 1
    $\begingroup$ Hehe. When there are two mutually exclusive and also true statements then there must be a third statement that resolves them. If $ D = 1-tanh^2(x)$ and $ D = sech^2(x)$ then $tanh^2(x) + sech^2(x) = 1$. It is about 1/3 the way down this page (en.wikipedia.org/wiki/Hyperbolic_function). Neither of these are the quadratic in the first DSIGMOID. "(0.666666667/1.7159*(1.7159-(S))*(1.7159+(S)))" Why is that statement correct? Within a radius of about 2-miles the flat-world hypothesis is a good approximation to the sphere of the earth. Counterexamples disprove, examples don't prove. $\endgroup$
    – EngrStudent
    May 30, 2013 at 1:48
  • $\begingroup$ Not a good idea to be dogmatic when you don't know key facts. $\endgroup$
    – Nick Cox
    May 30, 2013 at 6:29
-1
$\begingroup$

FANN_SIGMOID_SYMMETRIC

f2M_set_act_function_hidden (ann, FANN_SIGMOID_SYMMETRIC);

f2M_set_act_function_output (ann, FANN_SIGMOID_SYMMETRIC);

Symmetric sigmoid activation function, AKA tanh. One of the most used activation functions.

This activation function gives output that is between -1 and 1.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.