I have a simple neural network and it works with the logistic function as activation function. Now I want to avoid the saturation problem by substituting the logistic function by the hyperbolic tangent:
#define SIGMOID(x) (1.7159*tanh(0.66666667*x))
#define DSIGMOID(S) (0.666666667/1.7159*(1.7159-(S))*(1.7159+(S)))
But the network never converges, the MSE stays the same throughout the training. Here's my training samples:
double training_data[][4]={
{0, 0, 0, -1},
{0, 0, 1, 1},
{0, 1, 0, 1},
{0, 1, 1, -1},
{1, 0, 0, 1},
{1, 0, 1, -1},
{1, 1, 0, -1},
{1, 1, 1, 1}};
The network does converge if I use the original (non-scaled) hyperbolic tangent function, that is:
#define SIGMOID(x) (tanh(x))
#define DSIGMOID(S) (1-((S)*(S)))
Do I miss something? E.g. Scaling the output to match the range (-1.7159, 1.7159) or anything?