In neural networks back propagation we are trying to minimise our cost function W.R.T our parameters ( theta) . We penalize our neural network for every wrong prediction and don't penalize when predictions are right. In multi class where we are trying to classify the outputs into more than 2 categories (y is greater than 2).. The neural network basically gives an output of probability that a output is y is 3 or y is 4(example). So my neural network is predicting that y is 3 by .92(probability) and y is 4 is .78 (probability). actually the value output by neural network is correct ...it is indeed y=3. But does my neural network gets penalized for predicting y is 4 as .78? because in actual it is y is 3. so should their be any penalty for predicting .78 for y is 4 ?
this is a general situation for illustration...dont ask for the code of above situation.