1
$\begingroup$

I'm somewhat familiar with parametric estimation using MLE in the context of fitting the parameters of a distribution given a sample. Is there a way of generalizing this approach to unnormalized models (for instance, neural networks)? Naïvely maximizing the predicted log-likelihood would simply lead to the model predicting high values everywhere.

$\endgroup$
1
  • $\begingroup$ Can you add more information about you problem? Ideally, provide a mathematical description of the problem. Even a concrete example would clarify what you're asking. $\endgroup$ – Eli Dec 26 '20 at 22:27
1
+50
$\begingroup$

You are right that naive maximization of the likelihood would lead to blindly predicting high values.

You should look into Energy-based models (EBMs). EBMs define a likehood out of an arbitrary function $E_\theta(x)$ ("the energy function"): $$p_\theta(x)=\frac{\exp(−E_\theta (x))}{Z(\theta)}$$ where $x$ is an input, $\theta$ is the model's parameters (e.g., neural network weights) and $Z(\theta)$ is the partition function ($\int_x \exp(-E_\theta(x))$, which is not explicitly defined by the model.

The training of EBMs aims to choose a parameter set $\theta$ that would maximize $p_\theta(X)$ for the training distribution. This is achieved by using sampling methods to approximate $Z(\theta)$ during training, so we can train the model as if we had $Z(\theta)$ defined. The end result (ideally) is a model that maps the input $x$ to an output $E_\theta(x)$ which can be read out as an unnormalized negative-log-likelihood.

References:

https://en.wikipedia.org/wiki/Energy_based_model

LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., & Huang, F. (2006). A tutorial on energy-based learning. Predicting structured data, 1(0).

Grathwohl, Will, et al. "Your classifier is secretly an energy based model and you should treat it like one." arXiv preprint arXiv:1912.03263 (2019).

$\endgroup$
2
  • $\begingroup$ Exactly what I was looking for, thanks! If I may ask a stupid question: what is the difference between the normalization constant and the partition function? Is it simply a question of terminology? $\endgroup$ – yawn Dec 27 '20 at 5:42
  • 1
    $\begingroup$ My understanding is that the partition function is a particular case of a normalization constant (see en.wikipedia.org/wiki/Boltzmann_distribution). $\endgroup$ – Trisoloriansunscreen Dec 27 '20 at 6:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.