1
$\begingroup$

Suppose that we have have scoring functions $f(\textbf{x})$ and $g(\textbf{x})$ for classifying an object as red or blue. These are based on linear discriminant analysis. So if $f(\textbf{x}) > g(\textbf{x})$, then the object is red. Why is the posterior probability that an object is red the following:

$$\frac{\exp(f(\textbf{x})}{\exp(f(\textbf{x}))+\exp(g(\textbf{x}))}$$

Why do we exponentiate the functions?

$\endgroup$
  • 2
    $\begingroup$ Can you provide some references where the expression comes from? Are you doing Fischer LDA, or Canonical LDA? f and g cannot just be "some" scoring functions, otherwise the expression wouldn't be a valid probability. So please state the prior, the likelihood, as well as what f(x) is, or provide some references where the term occurs. $\endgroup$ – means-to-meaning Nov 4 '13 at 22:05
  • $\begingroup$ You might also want to look at Gibbs measure and how the energy of a configuration and its probability are related. $\endgroup$ – Luca Mar 6 '14 at 11:59
2
$\begingroup$

A good explanation is given in LeCunn's paper A tutorial on Energy Based Learning. The paper provides a detailed taxonomy of different score functions and losses, and their properties.

The expression you give has the following canonical form, $$ P(Y|X) = \frac{e^{\beta E(Y,X)}}{\sum_{Y}e^{\beta E(Y,X)}} $$ where $E(Y,X)$ the energy functions (what you referred to as scoring functions), are a measure of how plausible is to assign a sample $X$ the label $Y$. Under this transformation, one can give a probabilistic interpretation to finding the optimal label. Finding the $Y$ that minimizes the energy (inference) maximizes at the same time the likelihood.

This is relevant when combining different models. In such case, energy functions might fall in different ranges, so one needs to normalize them in a consistent way. Also, it provides a score of how confident it is that the returned label is the correct one compared to all other possibilities (notice the sum in the denominator).

It has also nice properties related to the quality of the learned model. The downside is that it is computationally very hard to solve the optimization method ought to the denominator of the expression.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.