For classification, one of the most common loss functions for artificial neural networks (ANN) is cross-entropy. What about in ANN for regression?
and why is cross-entropy hardly even discussed in ANN's encyclopedia entries?
$\begingroup$
$\endgroup$
7
-
$\begingroup$ Depends on the implementation, probably MSE or one of it's variants. $\endgroup$– user2974951Commented Aug 31, 2020 at 11:48
-
$\begingroup$ Cross entropy is one of the advised loss functions for classification (not dictated) $\endgroup$– gunesCommented Aug 31, 2020 at 11:50
-
$\begingroup$ what are the dictated loss functions for ANN regression. and what are the advised loss functions for ANN regression $\endgroup$– develaristCommented Aug 31, 2020 at 11:54
-
4$\begingroup$ no one can dictate a loss, it's your ANN. typical losses are mse, mae etc. $\endgroup$– gunesCommented Aug 31, 2020 at 11:57
-
2$\begingroup$ why did you think cross entropy loss for general regression in the first place? $\endgroup$– gunesCommented Aug 31, 2020 at 12:14
|
Show 2 more comments
2 Answers
$\begingroup$
$\endgroup$
4
The common loss function for regression with ANN is quadratic loss (least squares). If you're learning about NN from popular online courses and books, then you'll be told that classification and regression are two common kinds of problems where NN are applied. With the former being to fit to continuous output Y, and the latter for the categorical Y such as man/woman. So, in the regression least squares are quite popular, and they make no sense in many classification applications.
However, cross-entropy loss is used in regression too. For instance, when predicting the probability of default of loans the labels are 0 (current) and 1 (default) for historical observations of defaults in the loan pool. So this will look like a classification problem, but it really is a logit regression, where you're forecasting default probability (PD) of a loan pool using logistic activation in the output layer and cross-entropy loss. The trick is that PD of a pool is a continuous quantity, while the loan state observations are clearly categorical "default" or "current".
Why would you start with least squares and not cross-entropy? The reason is that least squares is the foundational concept, while cross-entropy is somewhat more advanced. Look, I can explain least squares to an elementary school student. I can't do the same with "entropy." In fact, most people who use cross-entropy do not have a slightest clue of what is entropy.
-
$\begingroup$ The point about cross-entropy to predict a probability doesn't make sense to me for a regression problem where the true observations span a continuum rather than discrete categories. $\endgroup$– DaveCommented Aug 31, 2020 at 16:03
-
$\begingroup$ @Dave just think of a logit regression $\endgroup$– AksakalCommented Aug 31, 2020 at 16:13
-
1$\begingroup$ The trouble with a logistic regression is that, while we have predictions on a continuum, the true observations are discrete: yes/no, dog/cat, etc. I also think I object to you saying that least squares makes no sense in classification. What about Brier score? $\endgroup$– DaveCommented Aug 31, 2020 at 16:18
-
$\begingroup$ @Dave, "you saying that least squares makes no sense in classification" - never said that, read my answer carefully. the same goes to your "cats/dogs" example, read the answer again $\endgroup$– AksakalCommented Aug 31, 2020 at 16:25
$\begingroup$
$\endgroup$
What about in ANN for regression?
Your intuition is correct that neural networks are not solely limited to classification problems, even though classification is a very common application.
There are lots of regression loss functions:
- Minimize square error ($L^2$ loss), which is the cross-entropy for a target modeled as having a Gaussian distribution;
- Minimize absolute error ($L^1$ loss), which is the cross-entropy for a target modeled as having a Laplace distribution;
- Minimize log cosh loss, which is a kind of compromise between $L^2$ and $L^1$ losses. See: When is Log-Cosh Loss used?
- And many more! A nice thing about modern neural networks software is that it's designed to allow customized models, including choosing from among many loss functions, or even allowing users to define their own losses.
and why is cross-entropy hardly even discussed in ANN's encyclopedia entries?
I can't speak to why Wikipedia makes editorial decisions.
A key terminology note is that "cross-entropy" is technically ambiguous. In a neural network setting, it's common to use the jargon "cross-entropy" to mean "binary or multinomial cross-entropy." However, if we're being technically precise, then cross-entropy is a general way of writing down loss functions which can be described in terms of a specific probability model. For more information and some examples, see How to construct a cross-entropy loss for general regression targets?
Another terminology note is that it's common to distinguish between "regression" and "classification," but there is a specific sense in which this distinction is spurious. Consider a logistic regression. It's common to speak of this as a "classification" model even though what's really happing is that we're creating a regression against the probabilities of the binomial/multinomial outcome. See: Why isn't Logistic Regression called Logistic Classification?
As a matter of jargon, it's common enough to use "classification" to mean "this model has a categorical outcome." But it's best not to get too attached to jargon -- think of it more as a short-hand for deeper technical concepts.
