It's common short-hand in neural networks literature to refer to categorical cross-entropy loss as "cross-entropy," even though there are a number of loss functions which could properly be described that way.
So, in general, how does one move from an assumed probability distribution for the target variable to defining a cross-entropy loss for your network? What does the function require as inputs? (For example, the categorical cross-entropy function for one-hot targets requires a one-hot binary vector and a probability vector as inputs.)
A good answer will discuss the general principles involved, as well as worked examples for
- categorical cross-entropy loss for one-hot targets
- Gaussian-distributed target distribution and how how this reduces to usual MSE loss
- A less common example such as a gamma distributed target, or a heavy-tailed target
- Explain the relationship between minimizing cross entropy and maximizing log-likelihood.
Inspired by Tensorflow Cross Entropy for Regression? with thanks to CowboyTrader.