0
$\begingroup$

This might be more of a math question, but since it concerns ML, I'll ask it here.

In "A tutorial on energy-based learning" (LeCun et al., 2006), on page 15, section 2.2.4 about the Negative Log-Likelihood Loss, is written: "Interestingly, the NLL loss reduces to the generalized perceptron loss when $\beta \to \infty $ (zero temperature),...".

The $\beta$ denotes the inverse temperature in the following formulation of the NLL, for a single sample, with $E$ an energy function and with the Gibbs distribution (wikipedia) filled in for $ P(Y^i \mid X^i, W) $:

$$ L = E(W, Y^i, X^i) + \frac{1}{\beta} \, \log \int_y \exp (- \beta E(W, y, X^i) $$

The generalized perceptron loss is defined as $ L(Y^i, (W, \mathcal{Y}, X^i)) = E(W, Y^i, X^i) - \min_{y \in \mathcal{Y}} E(W, y, X^i) $. So, the statement amounts to saying:

$$ \lim_{\beta \to \infty} \big( \frac{1}{\beta} \, \log \int_y \exp (- \beta E(y) \big ) \ = \ - \min_{y \in \mathcal{Y}} E(y) $$

Where $ E(Y^i) $ is actually $ E(W, Y^i, X^i) $, but shortened for clarity. I'm having trouble understanding exactly why. I've tried two roads. The first one is applying l'Hospital's rule for $\frac{\infty}{\infty}$ to the LHS, which (if I didn't make mistakes) gives:

$$ \lim_{\beta \to \infty} \big( \frac{1}{\beta} \, \log \int_y \exp (- \beta E(y)) \big ) \ = \ - \int_y E(y) \lim_{\beta \to \infty} \frac{\exp(- \beta E(y))}{ \int_y \exp(- \beta E(y)) } $$

The fraction in the integral is a probability (Gibbs distributed), so the entire integral seems like some kind of mean value of $E$ over $\mathcal{Y}$ ? Only for $\beta \to \infty$, which I don't know how to interpret. Applying l'Hospital's rule again is not allowed (I think?) because the limit of the derivative of the denominator $= 0$, and it doesn't bring me any further anyway, and neither does substituting $\alpha$ for $\exp(-\beta)$. The second way of starting was inspired by the generalized mean (wiki), which approaches the maximum for $p \to \infty$:

$$ \lim_{\beta \to \infty} \big( \frac{1}{\beta} \, \log \int_y \exp (- \beta E(y) \big ) \\ = \log \, \lim_{\beta \to \infty} \big( \int_y \exp(-E(y)) ^ \beta \big)^{1/\beta} \\ = \log \, \lim_{\beta \to \infty} \, \exp(- \min_y E(y)) \big( \int_y \big( \frac{\exp(-E(y))}{\exp(- \min_y E(y))} \big)^{\beta} \big)^{1/\beta} $$ If the last integral were a discrete sum over all $ y \in \mathcal{Y}$, all terms would be $<1$ and hence go to $0$ for $\beta \to \infty$, except for the term were $E(y) = min_y E(y)$, which would go to $1$, and the expression would then become:

$$ \log \, \exp( -\min_y E(y)) \, \lim_{\beta \to \infty} (...) \\ = \log \, \exp( -\min_y E(y)) \\ = - \min_{y \in \mathcal{Y}} E(y) $$

Which is what we were looking for. However, I don't know if extracting $ \exp( -\min_y E(y)) $ out of the integral like I'd do with a sum is correct, and if the limit would then equal $1$, as it would for a sum. Could someone perhaps explain this?

Was applying l'Hospital's rule correct? Could we have gotten to the result following that way of starting as well? And could someone shed some light on what happens with the Gibb's distribution when $\beta \to \infty$, or, as in physics, when $T \to 0$? I can't find a clear and concise explanation on internet.

Many thanks in advance :).

$\endgroup$
0
$\begingroup$

With some kind help from https://math.stackexchange.com/questions/3671030/negative-log-likelihood-loss-with-gibbs-distribution-for-beta-approaching-infini :

Assuming that $\exp(−E(\cdot))$ is in $𝐿^\infty$, you can use that $\lim_{\beta \to \infty}\Vert\exp(−E(\cdot))\Vert_\beta = \Vert\exp(−𝐸(⋅))\Vert_\infty$.

Also explained here: https://en.wikipedia.org/wiki/Uniform_norm , and proven here https://math.stackexchange.com/questions/242779/limit-of-lp-norm . The idea of the proof is similar to my second approach.

A sufficient condition for $\exp(−E(\cdot))$ to be in $𝐿^\infty$ is that $E$ is bounded.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.