Cost Function (Neural Network) for Forward Propagation

Question

This question is related to Andrew Ng's machine learning course on Coursera. Basically, when I calculate the cost function of a neural network, I use the following formula that was described by Ng: $$ J(\theta) = \frac 1m \sum_{i=1}^m\sum_{k=1}^k\begin{bmatrix}-y_k^{(i)}\log((h_\theta(x^{(i)}))_k) - (1-y_k^{(i)})\log(1-(h_\theta(x^{(i)}))_k)\end{bmatrix} $$

Let's call: \begin{align} j_1 &= -y_k^{(i)}\log((h_\theta(x^{(i)}))_k)\\j_2&= - (1-y_k^{(i)})\log(1-(h_\theta(x^{(i)}))_k)\\j&= j_1+j_2 \end{align} So the exercise asked me to convert any $y$ and $h_\theta(x)$ matrix into a binary matrix before performing the computation. Now this is when I don't understand...either I am getting it wrong or I am misunderstanding a concept.

If $y_k=0$ and $h_\theta(x)=0$ then $j_1=0, j_2=0, j=0$. Similarly

If $y_k=1$ and $h_\theta(x)=1$ then $j_1=0, j_2=0, j=0$.

Which seems fair since there should be no impact to the overall cost function.

How about...

If $y_k=1$ and $h_\theta(x)=0$ then $j_1$ is undefined since i cannot perform $\log(0)$. ? and also

If $y_k=0$ and $h_\theta(x)=1$ then I will also face the $\log(0)$ problem?

I think I am interpreting something wrongly but am not sure...

Answer 1

The short answer is that for classifier networks, an output activation is normally used that restricts $h_\theta(x)$ to the open interval (0, 1), so you never end up computing these $\log(0)$ values. Read on for details!

This loss is known as the categorical cross-entropy. It measures the difference between the true distribution over labels, and the distribution generated by the neural network. This loss is usually used (instead of the mean-squared error) for a classifier network because it has nice gradient properties.

Often, in a network used for classification in this way, the output layer uses a softmax activation to get the final $h_\theta(x)$ value: $$ g(\vec{z}) = \frac{\exp(\vec{z})}{\sum_i\exp(z_i)}. $$ This has several benefits:

• it normalizes the output values to sum to 1,
• it bounds the output values in practice to the open interval (0, 1).

To generate a 0, for instance, the pre-activation input on a unit would have to be a negative number large enough to cause exp(x) to overflow to 0. Likewise, to generate a 1 output, the outputs of all other units would have to be zero, which is nearly impossible for the same reason.

So it's basically safe to compute $\log(h_\theta(x))$ and $\log(1 - h_\theta(x))$ in the cross-entropy loss because in practice the $h_\theta(x)$ values are limited to the open interval (0, 1).