Many of the losses used in regression/classification tasks correspond to maximum likelihood estimation (MLE) or maximum aposteriori (MAP) under a specific data likelihood distribution $p(\mathbf{y}|X,\mathbf{w})$ and a possible prior $p(\mathbf{w})$. For example:

  • Log loss <-> sigmoid likelihood
  • Weighted Least squares <-> Gaussian likelihood with different $\sigma_i$
  • L1 regression <-> Laplace likelihood
  • Lasso <-> Gaussian likelihood with Laplace prior
  • Hinge Loss <-> [1]
  • ...

Intuitively every likelihood can be used to construct a loss function. Is the opposite true? I.e. do all loss correspond to a likelihood distribution? Note that the question is theoretical/philosophical in the sense that the distribution might be extremely complicated (not continuous / using delta functions / ...).

Of course, the question depends on the definition of a loss function. The least restrictive would be any function $L(\mathbf{y},\hat{\mathbf{y}}) \geq 0, \ \forall \mathbf{y},\hat{\mathbf{y}}$. I was wondering whether there might be conditions on the loss function $L$ that would make this true, similar to how positive-semidefinite kernels correspond to dot products in a different space. Intuitively I would say yes :).

[1] Franc, V., Zien, A., & Schölkopf, B. (2011, June). Support Vector Machines as Probabilistic Models. In ICML (pp. 665-672).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.