# Cross Entopy Loss for classification

Suppose I have a neural network, which classifies pictures of cats, dogs and fishes. The neural network uses Softmax as an activation function of the output layer.

Let's say I feed a picture of dog and the output I got is:

Cat : $$0.09$$ Dog : $$0.90$$ Fish : $$0.01$$

If I calculate the loss using Cross-Entropy I got:

$$-\sum_{c=1}^3y_{c}*\ln(\hat{y_{c}}) = -(0 * \ln(0.09) + 1 * \ln(0.90) + 0 * \ln(0.01)) = 0.105$$

However I would get the same loss value, if the output of my network were:

Cat : $$0.01$$ Dog : $$0.90$$ Fish : $$0.09$$

or

Cat : $$0.05$$ Dog : $$0.90$$ Fish : $$0.05$$

In another words, if I feed the picture of a dog the only thing that Cross-Entropy cares about is the output of the dog's neuron.

It seems to me, that if the network assigns $$.09%$$ probability to a fish and $$.01%$$ to a cat, while the correct answer is dog, is much worse than assigning $$.09$$ to cat and $$.01%$$ to fish, however the Cross Entropy does not penalize for that.

If I use Mean Squared Error, the loss of the network would depend on the output of each neuron from the output layer, for these reasons would it be better to use MSE instead of Cross Entropy Loss in my example ?