1
$\begingroup$

It's common short-hand in neural networks literature to refer to categorical cross-entropy loss as "cross-entropy," even though there are a number of loss functions which could properly be described that way.

So, in general, how does one move from an assumed probability distribution for the target variable to defining a cross-entropy loss for your network? What does the function require as inputs? (For example, the categorical cross-entropy function for one-hot targets requires a one-hot binary vector and a probability vector as inputs.)

A good answer will discuss the general principles involved, as well as worked examples for

  • categorical cross-entropy loss for one-hot targets
  • Gaussian-distributed target distribution and how how this reduces to usual MSE loss
  • A less common example such as a gamma distributed target, or a heavy-tailed target
  • Explain the relationship between minimizing cross entropy and maximizing log-likelihood.

Inspired by Tensorflow Cross Entropy for Regression? with thanks to CowboyTrader.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.