This question is related to Andrew Ng's machine learning course on Coursera. Basically, when I calculate the cost function of a neural network, I use the following formula that was described by Ng: $$ J(\theta) = \frac 1m \sum_{i=1}^m\sum_{k=1}^k\begin{bmatrix}-y_k^{(i)}\log((h_\theta(x^{(i)}))_k) - (1-y_k^{(i)})\log(1-(h_\theta(x^{(i)}))_k)\end{bmatrix} $$

Let's call: \begin{align} j_1 &= -y_k^{(i)}\log((h_\theta(x^{(i)}))_k)\\j_2&= - (1-y_k^{(i)})\log(1-(h_\theta(x^{(i)}))_k)\\j&= j_1+j_2 \end{align} So the exercise asked me to convert any $y$ and $h_\theta(x)$ matrix into a binary matrix before performing the computation. Now this is when I don't understand...either I am getting it wrong or I am misunderstanding a concept.

If $y_k=0$ and $h_\theta(x)=0$ then $j_1=0, j_2=0, j=0$. Similarly

If $y_k=1$ and $h_\theta(x)=1$ then $j_1=0, j_2=0, j=0$.

Which seems fair since there should be no impact to the overall cost function.

How about...

If $y_k=1$ and $h_\theta(x)=0$ then $j_1$ is undefined since i cannot perform $\log(0)$. ? and also

If $y_k=0$ and $h_\theta(x)=1$ then I will also face the $\log(0)$ problem?

I think I am interpreting something wrongly but am not sure...


1 Answer 1


The short answer is that for classifier networks, an output activation is normally used that restricts $h_\theta(x)$ to the open interval (0, 1), so you never end up computing these $\log(0)$ values. Read on for details!

This loss is known as the categorical cross-entropy. It measures the difference between the true distribution over labels, and the distribution generated by the neural network. This loss is usually used (instead of the mean-squared error) for a classifier network because it has nice gradient properties.

Often, in a network used for classification in this way, the output layer uses a softmax activation to get the final $h_\theta(x)$ value: $$ g(\vec{z}) = \frac{\exp(\vec{z})}{\sum_i\exp(z_i)}. $$ This has several benefits:

  • it normalizes the output values to sum to 1,
  • it bounds the output values in practice to the open interval (0, 1).

To generate a 0, for instance, the pre-activation input on a unit would have to be a negative number large enough to cause exp(x) to overflow to 0. Likewise, to generate a 1 output, the outputs of all other units would have to be zero, which is nearly impossible for the same reason.

So it's basically safe to compute $\log(h_\theta(x))$ and $\log(1 - h_\theta(x))$ in the cross-entropy loss because in practice the $h_\theta(x)$ values are limited to the open interval (0, 1).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.