What is the loss function used for CNN?

Question

For example, in AlexNet, they never specified what loss function they were using.

https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf

The output is a probability vector of dimension $1000 \times 1$

So either they are using Euclidean distance with one-hot encoded cateogries.

Or some multi-class logistic regression loss.

Can someone help?

Answer 1

In most cases CNNs use a cross-entropy loss on the one-hot encoded output. For a single image the cross entropy loss looks like this:

$$ - \sum_{c=1}^M{(y_c \cdot \log{\hat y_c})} $$

where $M$ is the number of classes (i.e. $1000$ in ImageNet) and $\hat y_c$ is the model's prediction for that class (i.e. the output of the softmax for class $c$). Due to the fact that the labels are one-hot encoded and $y$ is a $(1000 \times 1)$ vector of ones and zeroes, $y_c$ is either $1$ or $0$. Thus, out of the whole sum only one term will actually be added: the one with $y_c=1$.

Answer 2

As Jan says in a comment, AlexNet uses cross entropy as the loss function.

It's important to note, though, that a Convolutional Neural Network describes the architecture of the network, not the goal of the network. It is the goal of a network that determines the loss function.

CNN architectures can be used for many tasks with different loss functions:

• multi-class classification as in AlexNet
  • Typically cross entropy loss
• regression
  • Typically Squared Error loss
• image segmentation
  • Can use cross entropy loss as well, but can also use several other kinds of loss functions
• reinforcement learning
  • In Deep Q-Networks, the "Expected discounted accumulated future reward" can be used
• generative adversarial networks (generating images)
  • The Jensen–Shannon divergence was used in the original implementation