How does one choose a loss function for a given problem? (I've looked through stackexchange, and I haven't been able to find a thread that discusses this.)

Let say I observe some data $x \in \mathbb{R}^n$, and I'm interested in estimating some parameter $\theta$ related to the distribution from which $x$ came. Suppose that $\theta$ lies in an unbounded parameter space. Suppose that I'm interested in an estimator of the form: $\hat\theta = \arg\min_\theta \mathcal{L}(x, \theta)$, where $\mathcal{L}$ is some loss function. Is there a principled way to choose $\mathcal{L}$?

Some comments:

  • One way to choose the loss function would be through a probabilistic modeling approach. For instance, using squared error in regression could be motivated as being naturally derived from the likelihood of a gaussian with spherical covariance. However, this loss function has received some justification in this context beyond normality assumptions from the Gauss-Markov theorem.
  • The loss functions I've seen for GLMs are also derived from probabilistic considerations. However, we know that the estimated coefficient is robust in that it will still be consistent even if the distribution is misspecified (as long as the link is correctly specified and all of true covariates are included.) (See Gourieroux, C., A. Monfort, and A. Trognon. 1984. Pseudo maximum likelihood methods: theory. Econometrica 52: 681–700.) This provides some (asymptotic) robustness to the choice of loss function.
  • When doing nonnegative matrix factorization, you can consider the negative likelihood when assuming that the matrix entries are drawn from a Poisson distribution. However, we can show that the loss function derived from this is equivalent to (generalized) kullback-leibler divergence. See, for instance, Relationship between Poisson generation and generalized Kullback-Leibler divergence. This is nice since it provides an information-theoretic motivation for the loss function.
  • The loss function could be chosen by practical considerations such as computational efficiency. Surely squared error has received such mileage for this reason.
  • Perhaps, if the situation calls for such a property, the loss function can be derived in an ad-hoc way by introducing asymmetry into a symmetric loss function.
  • In machine learning, objective functions are often used that are not derived from probabilistic modeling; however, I'm not sure if those objective functions should be considered loss functions.

Is there some theory concerning the choice and effect of loss function?

I'm also interested in the choice of loss function when determining admissibility of a given estimator. So, in this setting, an estimator is just given, and we're interested in determining whether the risk (or expected loss) of the estimator is ever dominated by another estimator. Is there a framework for choosing a loss function in this setting that's different in the setting where we use the loss function to produce an estimator?


Canonically your loss function should reflect the losses you sustain from forecasting error. For instance, when your weather forecast tells you there's no rain today, while in fact it rains, you call a taxi to get home instead of walking and it costs you \$10. Alternatively, the forecast projected rain, so you drove to work instead of walking, which cost you \$2 of gas. So, your loss function is (false rain: 1, false clear: 5)

In a real world people don't bother to construct the loss functions of their problems, but use one of the default choices such as RMSE or MAE. Why? Mainly, because they don't know better, but also because it's "difficult".

Note, that the loss function is not related to probabilities or likelihoods at all. You minimize the probability weighted loss, i.e. each loss will be linked to the probability, then the expected loss is minimized.

  • $\begingroup$ Thanks for responding! I agree that in very simple cases where the response can only be yes/no, the loss function choice simplifies greatly. However, your last paragraph it unclear to me. What does " the loss function is not related to probabilities or likelihoods at all" mean. In my first and second (and third) bulleted point I hope to convey that this is a typical way to construct loss functions. $\endgroup$ – user795305 Aug 10 '17 at 18:50
  • $\begingroup$ For instance, if $y \sim \mathcal{N} (X \beta^*, \sigma^2 I)$, then maximizing the likelihood of $y$ is equivalent to minimizing least squares $\|y - X \beta\|_2^2$. That is, probabilistic considerations motivate the loss function. $\endgroup$ – user795305 Aug 10 '17 at 18:50
  • $\begingroup$ No, the loss function is not motivated by probabilities. The expected loss is of course driven by both probabilities and the loss function. In my example there are no probabilities when coming with \$2 or \$10 losses. These are the states of nature and you look at the outcomes given that you're in this state. The probabilities are attached to the states of nature, but outcomes do not depend on them. $\endgroup$ – Aksakal Aug 10 '17 at 19:21
  • $\begingroup$ Ah, okay! I attempted in the beginning of my first comment to draw attention way from binary outcome situations. Towards this, I just edited the question to specify that the parameter of interest lies in an unbounded parameter space. I'm sorry for the confusion! In light of this edit, I hope you reconsider my second comment. I suspect you'll find that many loss functions usually encountered have probabilistic underpinnings/motivations. $\endgroup$ – user795305 Aug 10 '17 at 19:28
  • $\begingroup$ The binary loss function that I brought up was just a toy example. You can and should construct appropriate loss functions whether bounded or not, continues or discrete etc. You'll see why you can't motivate the loss function by probabilities, if you think of what's a random variable is. When you construct a random variable, you never talk about its probability (measure), you only map the numbers to events space. Or even simpler, consider this: what if you changed your model, and now all your error probabilities are different? Are you going to "adjust" your loss function now? $\endgroup$ – Aksakal Aug 10 '17 at 19:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.