Cross Entopy Loss for classification

Question

Suppose I have a neural network, which classifies pictures of cats, dogs and fishes. The neural network uses Softmax as an activation function of the output layer.

Let's say I feed a picture of dog and the output I got is:

Cat : $0.09$ Dog : $0.90$ Fish : $0.01$

If I calculate the loss using Cross-Entropy I got:

$$ -\sum_{c=1}^3y_{c}*\ln(\hat{y_{c}}) = -(0 * \ln(0.09) + 1 * \ln(0.90) + 0 * \ln(0.01)) = 0.105 $$

However I would get the same loss value, if the output of my network were:

Cat : $0.01$ Dog : $0.90$ Fish : $0.09$

or

Cat : $0.05$ Dog : $0.90$ Fish : $0.05$

In another words, if I feed the picture of a dog the only thing that Cross-Entropy cares about is the output of the dog's neuron.

It seems to me, that if the network assigns $.09%$ probability to a fish and $.01%$ to a cat, while the correct answer is dog, is much worse than assigning $.09$ to cat and $.01%$ to fish, however the Cross Entropy does not penalize for that.

If I use Mean Squared Error, the loss of the network would depend on the output of each neuron from the output layer, for these reasons would it be better to use MSE instead of Cross Entropy Loss in my example ?

Answer 1

With MSE, you have the same problem you described in your post.

Your intuition that mistaking a dog for a cat is less of an bad thing than mistaking a dog for a fish can be made part of the loss function by adding a cross-entropy term for whether the image is of a quadruped or not. You can use the sum of your estimated dog and cat probability as the network's prediction of quadruped or not.