In Andrew Ng's online machine learning course in the part about neural nets for classification the following convex cost function is given: $$\text{cost} = -y\log(h_0(x)) - (1-y)\log(1-h_0(x))$$ which is predicated on the output labels being either 1 or 0 (a sigmoid activation function). I would like to code a NN classifier using a hyperbolic tangent activation function, particularly that given in Le Cun 1998, shown below: $$1.7159\tanh((2/3)x)$$ which will take on values between >1 and < 0 in the limit.

Obviously the above cost function would not be suitable for this activation function, so could anyone let me know what sort of similar cost function for this hyperbolic tangent activation I should be looking for? So far my web search has been fruitless.

  • $\begingroup$ So what is your hypothesis function. How you decide a instance is class A in what case and with what confidence, and in Class B as same. $\endgroup$
    – erogol
    Commented Oct 16, 2012 at 23:10
  • $\begingroup$ @Erogol I will use labelled data to train the NN, with 1 being a positive case of the class and -1 being negative. $\endgroup$ Commented Oct 18, 2012 at 15:49
  • $\begingroup$ so what is the point to use a function that gives a value x <0 or >1 $\endgroup$
    – erogol
    Commented Oct 18, 2012 at 17:40

1 Answer 1


The cost function used with the sigmoid function was motivated by the maximum likelihood estimation, and $$\text{cost}=−y\log(h_0(x))−(1−y)\log(1−h_0(x))$$ is just another way of saying $$\text{cost}=−log(h_0(x))$$ when $y=1$ and $$\text{cost}=−\log(1−h_0(x))$$ when $y=0$.

Those motivations still exist no matter what the activation function is (sigmoid or hyperbolic tangent). I would map the hyperbolic function from the range (-1,1) to the same range as the sigmoid (0,1) so that:

$$\text{cost} = −\frac{y+1}{2} \log{\left(\frac{h_\theta(x)+1}{2}\right)}−(1− \frac{y+1}{2})\log\left(1−\frac{h_\theta(x)+1}{2}\right)$$

Where $$h_\theta = \tanh\left(\frac{2}{3}x \right)$$.

This will have a different gradient than the sigmoid. Good luck.

  • $\begingroup$ Hi. Why do you need the 2/3 x transformation? $\endgroup$ Commented Jan 25, 2016 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.