Penalization in multi class neural networks backpropagation 
the cost function for my neural network.
In neural networks back propagation we are trying to minimise our cost function W.R.T our parameters ( theta) . We penalize our neural network for every wrong prediction and don't penalize when predictions are right. In multi class where we are trying to classify the outputs into more than 2 categories (y is greater than 2).. The neural network basically gives an output of probability that a output is y is 3 or y is 4(example). So my neural network is predicting that y is 3 by .92(probability) and y is 4 is .78 (probability). actually the value output by neural network is correct ...it is indeed y=3.
But does my neural network gets penalized for predicting y is 4 as .78? because in actual it is y is 3. so should their be any penalty for predicting .78 for y is 4 ?
this is a general situation for illustration...dont ask for the code of above situation.
 A: In a comment, you say that each sample represents exactly one class. This means that you're modeling mutually exclusive events. This means that $y$ is a vector with exactly one value of 1 and the rest 0. So if you have $K=3$ classes, then all vectors $y$ look like $[1,0,0]$ or $[0,1,0]$ or $[0,0,1]$.
We'll consider the loss for $m=1$ samples and $y=[1,0,0]$.
$$\begin{align}
J(\theta)-L^2 \text{ regulariztion}&=
-\frac{1}{1}\sum_{i=1}^{m=1}\sum_{k=1}^K y_k^{(i)} \log (h_\theta(x^{(i)})_k) + (1-y_k^{(i)}) \log(1-h_\theta(x^{(i)})_k) \\
&= -[\log (h_\theta(x)_1)+0] \\&~~- [0+\log(1-h_\theta(x)_2] \\&~~- [0+\log(1-h_\theta(x)_3)]\\
&= -\log (h_\theta(x)_1) - \log(1-h_\theta(x)_2) - \log(1-h_\theta(x)_3)
\end{align}$$
(We can neglect the $L^2$ regularization for this demonstration because it does not include $h_\theta(x^{(i)})$, which is all we care about.)
You can show the same for the other possible $y$ vectors, and likewise for any $K \ge 2$.
Conclusion. The predictions of all classes are penalized because the loss involves all elements of the 3-element vector $h_\theta(x^{(i)})$.
