For example, in AlexNet, they never specified what loss function they were using.


The output is a probability vector of dimension $1000 \times 1$

So either they are using Euclidean distance with one-hot encoded cateogries.

Or some multi-class logistic regression loss.

Can someone help?

  • 2
    $\begingroup$ CNNs are a type of network defined by a characteristic architecture. This has nothing to do with the loss function used for training. Various loss functions can be used, depending on the problem. $\endgroup$ – user20160 Oct 24 '19 at 7:56
  • 2
    $\begingroup$ The paper does actually say which loss function they use: "Our network maximizes the multinomial logistic regression objective", which is exactly the cross-entropy. $\endgroup$ – Jan Kukacka Oct 24 '19 at 9:36

In most cases CNNs use a cross-entropy loss on the one-hot encoded output. For a single image the cross entropy loss looks like this:

$$ - \sum_{c=1}^M{(y_c \cdot \log{\hat y_c})} $$

where $M$ is the number of classes (i.e. $1000$ in ImageNet) and $\hat y_c$ is the model's prediction for that class (i.e. the output of the softmax for class $c$). Due to the fact that the labels are one-hot encoded and $y$ is a $(1000 \times 1)$ vector of ones and zeroes, $y_c$ is either $1$ or $0$. Thus, out of the whole sum only one term will actually be added: the one with $y_c=1$.

  • 2
    $\begingroup$ “In most cases” needs a citation. CNNs can be used for regressions, or for image segmentation, or for reinforcement learning, or any number of tasks which would not use cross entropy. The loss function is independent of the architecture! $\endgroup$ – kbrose Oct 24 '19 at 13:03
  • 2
    $\begingroup$ @kbrose I said "in most cases" just to be modest. I have never seen a CNN for classification trained with anything other than cross entropy. $\endgroup$ – Djib2011 Oct 24 '19 at 19:14
  • $\begingroup$ Who said anything about classification? $\endgroup$ – kbrose Oct 25 '19 at 0:17
  • $\begingroup$ More thoroughly -- the question to me reads like it misunderstands the purpose of CNNs. CNNs are architectures that can be used for classification, but can be used for much more than that. I think an answer that doesn't address that fact is incomplete. That's what I was trying to say with my first comment. $\endgroup$ – kbrose Oct 25 '19 at 0:25
  • $\begingroup$ Yes you are correct and if you were just answering to the the question in the title, I would agree with you. But if you read the body of the question, you'll see he implies "classification". For someone whose beginning to learn about CNNs I think it is better to just answer his intended question than to create more questions by introducing more problem settings. $\endgroup$ – Djib2011 Oct 25 '19 at 6:35

As Jan says in a comment, AlexNet uses cross entropy as the loss function.

It's important to note, though, that a Convolutional Neural Network describes the architecture of the network, not the goal of the network. It is the goal of a network that determines the loss function.

CNN architectures can be used for many tasks with different loss functions:

  • multi-class classification as in AlexNet
    • Typically cross entropy loss
  • regression
    • Typically Squared Error loss
  • image segmentation
    • Can use cross entropy loss as well, but can also use several other kinds of loss functions
  • reinforcement learning
    • In Deep Q-Networks, the "Expected discounted accumulated future reward" can be used
  • generative adversarial networks (generating images)
    • The Jensen–Shannon divergence was used in the original implementation
  • 2
    $\begingroup$ From the link and from the output vector you can tell he is clearly referring to classification. $\endgroup$ – JkBk Oct 24 '19 at 22:31
  • $\begingroup$ I answered the title in the question. The body text of the question says “for example”, i.e. the OP is asking about the general case. I don’t know if it’s “clear” at all. $\endgroup$ – kbrose Oct 25 '19 at 0:16
  • $\begingroup$ I also answer the specific question in my very first sentence. $\endgroup$ – kbrose Oct 25 '19 at 0:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.