It's common short-hand in neural networks literature to refer to categorical cross-entropy loss as "cross-entropy," even though there are a number of loss functions which could properly be described that way.

So, in general, how does one move from an assumed probability distribution for the target variable to defining a cross-entropy loss for your network? What does the function require as inputs? (For example, the categorical cross-entropy function for one-hot targets requires a one-hot binary vector and a probability vector as inputs.)

A good answer will discuss the general principles involved, as well as worked examples for

  • categorical cross-entropy loss for one-hot targets
  • Gaussian-distributed target distribution and how how this reduces to usual MSE loss
  • A less common example such as a gamma distributed target, or a heavy-tailed target
  • Explain the relationship between minimizing cross entropy and maximizing log-likelihood.

Inspired by Tensorflow Cross Entropy for Regression? with thanks to CowboyTrader.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.