I'm somewhat familiar with parametric estimation using MLE in the context of fitting the parameters of a distribution given a sample. Is there a way of generalizing this approach to unnormalized models (for instance, neural networks)? Naïvely maximizing the predicted log-likelihood would simply lead to the model predicting high values everywhere.
1 Answer
You are right that naive maximization of the likelihood would lead to blindly predicting high values.
You should look into Energy-based models (EBMs). EBMs define a likehood out of an arbitrary function $E_\theta(x)$ ("the energy function"): $$p_\theta(x)=\frac{\exp(−E_\theta (x))}{Z(\theta)}$$ where $x$ is an input, $\theta$ is the model's parameters (e.g., neural network weights) and $Z(\theta)$ is the partition function ($\int_x \exp(-E_\theta(x))$, which is not explicitly defined by the model.
The training of EBMs aims to choose a parameter set $\theta$ that would maximize $p_\theta(X)$ for the training distribution. This is achieved by using sampling methods to approximate $Z(\theta)$ during training, so we can train the model as if we had $Z(\theta)$ defined. The end result (ideally) is a model that maps the input $x$ to an output $E_\theta(x)$ which can be read out as an unnormalized negative-log-likelihood.
References:
https://en.wikipedia.org/wiki/Energy_based_model
LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., & Huang, F. (2006). A tutorial on energy-based learning. Predicting structured data, 1(0).
Grathwohl, Will, et al. "Your classifier is secretly an energy based model and you should treat it like one." arXiv preprint arXiv:1912.03263 (2019).
-
$\begingroup$ Exactly what I was looking for, thanks! If I may ask a stupid question: what is the difference between the normalization constant and the partition function? Is it simply a question of terminology? $\endgroup$– yawnDec 27, 2020 at 5:42
-
1$\begingroup$ My understanding is that the partition function is a particular case of a normalization constant (see en.wikipedia.org/wiki/Boltzmann_distribution). $\endgroup$ Dec 27, 2020 at 6:21