4
$\begingroup$

For me a model (the way we calculate an output given an input) and a loss function (the way we estimate the accuracy of a model) always were two different (independent, "orthogonal") things. In other words, I though that any model can be trained and evaluated with any loss function.

However, it looks to me that there is some "binding" between models and loss functions. For example here I see the following statement:

You can also create your own loss function. Some examples of existing losses are: LinearRegressionOutput, which computes the l2-loss between it’s input symbol and the labels provided to it; SoftmaxOutput, which computes the categorical cross-entropy.

So, it looks like softmax function is somehow "bound" with the "cross-entropy" and linear regression is bound to the l2-loss. But what does this binding mean? In what way are they bound? I thought that softmax is just a way to normalize output (so that all the values are forced to sum up to one, so that they can be interpreted as probabilities). Is it incorrect to measure accuracy of softmax using squared deviations? Or, alternatively, can't we measure accuracy of linear regression by something other that l2-loss?

$\endgroup$
  • $\begingroup$ This binding thing is not an obligatory way of estimating model quality. "Linear Regression" associated with L2 -loss is just a regression task where You prefer to minimize squared residuals, and You prefer this over absolute residual loss. Likewise cross-entropy is just one (and it is known to be biased) measure of classification accuracy. You can extend the classification loss by at leats: accuracy, log-loss, and AUC for a binary case. It is all up to your design thinking. $\endgroup$ – Alexey Burnakov Oct 9 '17 at 14:11
  • $\begingroup$ I myself find it confusing to read function names like "Linear Regression Output" in place of the regression task performed on continuous output which not necessarily limits itself to linear relation only. Besides, consider another aspect to the problem, evaluation function, which will show you some scalar value after minimizing the Loss you defined, but not necessarily being the same loss measure. You can, for example, minimize Mean Squared Error, and ask the module to show you Root Mean Squared measure on the surface if it is the measure you welcome most. While MSE is differentiable. $\endgroup$ – Alexey Burnakov Oct 9 '17 at 14:16
  • $\begingroup$ @AlexeyBurnakov, this is what I thought as well. But what has confused me is the fact that in MXNet different models are rigidly bound with different loss functions. For example, if I use softmax in the output layer I am forced to use cross-entropy. I though that there should be a strong reason for that (like using any other measure would be fundamentally wrong). $\endgroup$ – Roman Oct 9 '17 at 14:17
  • $\begingroup$ In general, it is up to your understanding of math and your desire of using a given measure that you think suits best Your task. As I noted, your loss is expected to be differentiable, which you get by employing L2, not L1. But I don't get it quite well why cross entropy would be the default for multiclass. $\endgroup$ – Alexey Burnakov Oct 9 '17 at 14:20
  • $\begingroup$ @AlexeyBurnakov, to me it looks like "cross-entropy" is not just default for "soft-max" it is the only option for soft-max. They (MXNet) even use it interchangeable (if you want to use cross-entropy loss, you need to use soft-max output). $\endgroup$ – Roman Oct 9 '17 at 14:24
4
$\begingroup$

It may be important to define what a model is. For example the full model of linear regression is not just $Y=\beta X$ but:

  • $Y=\beta X+\epsilon$
  • $\epsilon$ has normal distribution with mean 0 and variance $\sigma$
  • $X$ and $\epsilon$ are independent

This defines totally the distribution of $Y$ given $X$ and you can define the likelihood $p(y|x,\beta,\sigma)$ of a single line, and then the likelihood of the full training set: $L(\beta,\sigma)=\Pi_i p(y_i|x_i,\beta,\sigma)$

There is special loss function derived from the likelihood $L$. It is simply $-\log(L)$. For linear regression this loss function happens to be OLS (modulo a few constants you don't care about and where $\sigma$ disappears magically). As a consequence, maximizing the likelihood for linear regression is the same as minimizing OLS.

In most models where $Y$ depends on $X$, a fully specified model is the distribution of $Y$ given $X$ (and a parameter). Thus every model has a likelihood. There is always a "canonical" way to create a loss function from a model: $-\log(L)$ (or any decreasing function instead of $-\log$ which does not change the solution of the minimization problem).

A very common way to fit models is to use maximum likelihood, hence implicitly minimize this loss function. It is called maximum likelihood estimation (MLE). Logistic regression is most often fitted with MLE. More generally, generalized linear models are fitted with MLE. Same for Gaussian mixtures...

But it is not the only way to do. If you think MLE is canonical, then yes, models come with a canonical loss function. But sometimes MLE is not used and another loss function can be used.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.