Skip to main content
Cross Validated

Return to Question

changed tag; formatted
Source Link
gung - Reinstate Monica
gung - Reinstate Monica
  • 147.5k
  • 89
  • 406
  • 716

I'm currently studying from Hinton's neural network course and he just introduced the cost function used with the softmax output function:

$$C=-\sum_{j}t_jlogy_j$$

$$y_j=\frac{e^{z_j}}{\sum_{k \in group}e^{z_k}}$$\begin{align} C &= -\sum_{j}t_j\log y_j \\[5pt] y_j &= \frac{e^{z_j}}{\sum_{k \in group}e^{z_k}} \end{align}

In the slides, he says that "C has a very big gradient when the target value is 1 and the output is almost zero". This makes sense as we will want to minimize $C$ since the target and output are not the same. However, what if the target value is 0? Here, wouldn't $C$ always be zero so it doesn't matter what weights you have in $y_j$?

I'm currently studying from Hinton's neural network course and he just introduced the cost function used with the softmax output function:

$$C=-\sum_{j}t_jlogy_j$$

$$y_j=\frac{e^{z_j}}{\sum_{k \in group}e^{z_k}}$$

In the slides, he says that "C has a very big gradient when the target value is 1 and the output is almost zero". This makes sense as we will want to minimize $C$ since the target and output are not the same. However, what if the target value is 0? Here, wouldn't $C$ always be zero so it doesn't matter what weights you have in $y_j$?

I'm currently studying from Hinton's neural network course and he just introduced the cost function used with the softmax output function:

\begin{align} C &= -\sum_{j}t_j\log y_j \\[5pt] y_j &= \frac{e^{z_j}}{\sum_{k \in group}e^{z_k}} \end{align}

In the slides, he says that "C has a very big gradient when the target value is 1 and the output is almost zero". This makes sense as we will want to minimize $C$ since the target and output are not the same. However, what if the target value is 0? Here, wouldn't $C$ always be zero so it doesn't matter what weights you have in $y_j$?

Source Link
jlcv
jlcv
  • 103
  • 3

Cost function for cross entropy

I'm currently studying from Hinton's neural network course and he just introduced the cost function used with the softmax output function:

$$C=-\sum_{j}t_jlogy_j$$

$$y_j=\frac{e^{z_j}}{\sum_{k \in group}e^{z_k}}$$

In the slides, he says that "C has a very big gradient when the target value is 1 and the output is almost zero". This makes sense as we will want to minimize $C$ since the target and output are not the same. However, what if the target value is 0? Here, wouldn't $C$ always be zero so it doesn't matter what weights you have in $y_j$?