Cost Function (Neural Network) for Forward Propagation This question is related to Andrew Ng's machine learning course on Coursera. Basically, when I calculate the cost function of a neural network, I use the following formula that was described by Ng:
$$
J(\theta) = \frac 1m \sum_{i=1}^m\sum_{k=1}^k\begin{bmatrix}-y_k^{(i)}\log((h_\theta(x^{(i)}))_k) - (1-y_k^{(i)})\log(1-(h_\theta(x^{(i)}))_k)\end{bmatrix}
$$
Let's call:
\begin{align}
j_1 &= -y_k^{(i)}\log((h_\theta(x^{(i)}))_k)\\j_2&= - (1-y_k^{(i)})\log(1-(h_\theta(x^{(i)}))_k)\\j&= j_1+j_2
\end{align}
So the exercise asked me to convert any $y$ and $h_\theta(x)$ matrix into a binary matrix before performing the computation. Now this is when I don't understand...either I am getting it wrong or I am misunderstanding a concept.
If $y_k=0$ and $h_\theta(x)=0$ then $j_1=0, j_2=0, j=0$. Similarly
If $y_k=1$ and $h_\theta(x)=1$ then $j_1=0, j_2=0, j=0$.
Which seems fair since there should be no impact to the overall cost function.
How about...
If $y_k=1$ and $h_\theta(x)=0$ then $j_1$ is undefined since i cannot perform $\log(0)$.
? and also
If $y_k=0$ and $h_\theta(x)=1$ then I will also face the $\log(0)$ problem?
I think I am interpreting something wrongly but am not sure...
 A: The short answer is that for classifier networks, an output activation is normally used that restricts $h_\theta(x)$ to the open interval (0, 1), so you never end up computing these $\log(0)$ values. Read on for details!
This loss is known as the categorical cross-entropy. It measures the difference between the true distribution over labels, and the distribution generated by the neural network. This loss is usually used (instead of the mean-squared error) for a classifier network because it has nice gradient properties.
Often, in a network used for classification in this way, the output layer uses a softmax activation to get the final $h_\theta(x)$ value: $$ g(\vec{z}) = \frac{\exp(\vec{z})}{\sum_i\exp(z_i)}. $$ This has several benefits:


*

*it normalizes the output values to sum to 1,

*it bounds the output values in practice to the open interval (0, 1).


To generate a 0, for instance, the pre-activation input on a unit would have to be a negative number large enough to cause exp(x) to overflow to 0. Likewise, to generate a 1 output, the outputs of all other units would have to be zero, which is nearly impossible for the same reason.
So it's basically safe to compute $\log(h_\theta(x))$ and $\log(1 - h_\theta(x))$ in the cross-entropy loss because in practice the $h_\theta(x)$ values are limited to the open interval (0, 1).
