cost function the cost function for my neural network.

In neural networks back propagation we are trying to minimise our cost function W.R.T our parameters ( theta) . We penalize our neural network for every wrong prediction and don't penalize when predictions are right. In multi class where we are trying to classify the outputs into more than 2 categories (y is greater than 2).. The neural network basically gives an output of probability that a output is y is 3 or y is 4(example). So my neural network is predicting that y is 3 by .92(probability) and y is 4 is .78 (probability). actually the value output by neural network is correct ...it is indeed y=3. But does my neural network gets penalized for predicting y is 4 as .78? because in actual it is y is 3. so should their be any penalty for predicting .78 for y is 4 ?

this is a general situation for illustration...dont ask for the code of above situation.

  • $\begingroup$ Can each sample belong to exactly 1 class, or can a sample have multiple classes? For example, if you're classifying images as "cat", "fish", "dog", does each image contain exactly one kind of animal, or are there images with "cat" and "fish", "cat" and "dog", etc.? What loss function are you using? $\endgroup$
    – Sycorax
    Sep 11, 2020 at 15:55
  • $\begingroup$ only 1 type.... in ur example $\endgroup$ Sep 11, 2020 at 16:37
  • $\begingroup$ thank u for ur time.. i have added the cost function $\endgroup$ Sep 11, 2020 at 17:10
  • $\begingroup$ What is $K$? What is $m$? $\endgroup$
    – Sycorax
    Sep 11, 2020 at 17:13
  • $\begingroup$ sorry .. K is the number of classes that my model is classifying into one out of K is correct. m is the number of training examples $\endgroup$ Sep 11, 2020 at 17:15

1 Answer 1


In a comment, you say that each sample represents exactly one class. This means that you're modeling mutually exclusive events. This means that $y$ is a vector with exactly one value of 1 and the rest 0. So if you have $K=3$ classes, then all vectors $y$ look like $[1,0,0]$ or $[0,1,0]$ or $[0,0,1]$.

We'll consider the loss for $m=1$ samples and $y=[1,0,0]$.

$$\begin{align} J(\theta)-L^2 \text{ regulariztion}&= -\frac{1}{1}\sum_{i=1}^{m=1}\sum_{k=1}^K y_k^{(i)} \log (h_\theta(x^{(i)})_k) + (1-y_k^{(i)}) \log(1-h_\theta(x^{(i)})_k) \\ &= -[\log (h_\theta(x)_1)+0] \\&~~- [0+\log(1-h_\theta(x)_2] \\&~~- [0+\log(1-h_\theta(x)_3)]\\ &= -\log (h_\theta(x)_1) - \log(1-h_\theta(x)_2) - \log(1-h_\theta(x)_3) \end{align}$$ (We can neglect the $L^2$ regularization for this demonstration because it does not include $h_\theta(x^{(i)})$, which is all we care about.)

You can show the same for the other possible $y$ vectors, and likewise for any $K \ge 2$.

Conclusion. The predictions of all classes are penalized because the loss involves all elements of the 3-element vector $h_\theta(x^{(i)})$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.