Many of the losses used in regression/classification tasks correspond to maximum likelihood estimation (MLE) or maximum aposteriori (MAP) under a specific data likelihood distribution $p(\mathbf{y}|X,\mathbf{w})$ and a possible prior $p(\mathbf{w})$. For example:

  • Log loss <-> sigmoid likelihood
  • Weighted Least squares <-> Gaussian likelihood with different $\sigma_i$
  • L1 regression <-> Laplace likelihood
  • Lasso <-> Gaussian likelihood with Laplace prior
  • Hinge Loss <-> [1]
  • ...

Intuitively every likelihood can be used to construct a loss function. Is the opposite true? I.e. do all loss correspond to a likelihood distribution? Note that the question is theoretical/philosophical in the sense that the distribution might be extremely complicated (not continuous / using delta functions / ...).

Of course, the question depends on the definition of a loss function. The least restrictive would be any function $L(\mathbf{y},\hat{\mathbf{y}}) \geq 0, \ \forall \mathbf{y},\hat{\mathbf{y}}$. I was wondering whether there might be conditions on the loss function $L$ that would make this true, similar to how positive-semidefinite kernels correspond to dot products in a different space. Intuitively I would say yes :).

[1] Franc, V., Zien, A., & Schölkopf, B. (2011, June). Support Vector Machines as Probabilistic Models. In ICML (pp. 665-672).


2 years later I'll give a partial answer, this encompasses the first three example (log loss, weighted log loss, L1 regression) and many more point wise losses.

Let $L : \mathcal{Y} \times \mathcal{Y}$ be a point-wise loss (i.e. scores a predicted $\hat y$ and $y$ by $L(\hat y,y)$). The essence would be to define the likelihood as:

$$ p(y|x) = \frac{1}{C} \exp (- L(y,x)) $$

for a suitable normalizing constant $C$ and assuming that the above equation is well defined. This is essentially what the authors of [1] did in proposition 2.9. But it essentially hides most of the complexity in whether that equation is well defined.

So the answer to the initial question would be no not all (point) losses gives rise to a well-defined density but many do.

[1] Gressmann, Frithjof, et al. "Probabilistic supervised learning." arXiv preprint arXiv:1801.00753 (2018).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.