How to fit a unnormalized parametric distribution with MLE?

I'm somewhat familiar with parametric estimation using MLE in the context of fitting the parameters of a distribution given a sample. Is there a way of generalizing this approach to unnormalized models (for instance, neural networks)? Naïvely maximizing the predicted log-likelihood would simply lead to the model predicting high values everywhere.

• Can you add more information about you problem? Ideally, provide a mathematical description of the problem. Even a concrete example would clarify what you're asking. – Eli Dec 26 '20 at 22:27

You are right that naive maximization of the likelihood would lead to blindly predicting high values.

You should look into Energy-based models (EBMs). EBMs define a likehood out of an arbitrary function $$E_\theta(x)$$ ("the energy function"): $$p_\theta(x)=\frac{\exp(−E_\theta (x))}{Z(\theta)}$$ where $$x$$ is an input, $$\theta$$ is the model's parameters (e.g., neural network weights) and $$Z(\theta)$$ is the partition function ($$\int_x \exp(-E_\theta(x))$$, which is not explicitly defined by the model.

The training of EBMs aims to choose a parameter set $$\theta$$ that would maximize $$p_\theta(X)$$ for the training distribution. This is achieved by using sampling methods to approximate $$Z(\theta)$$ during training, so we can train the model as if we had $$Z(\theta)$$ defined. The end result (ideally) is a model that maps the input $$x$$ to an output $$E_\theta(x)$$ which can be read out as an unnormalized negative-log-likelihood.

References:

https://en.wikipedia.org/wiki/Energy_based_model

LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., & Huang, F. (2006). A tutorial on energy-based learning. Predicting structured data, 1(0).

Grathwohl, Will, et al. "Your classifier is secretly an energy based model and you should treat it like one." arXiv preprint arXiv:1912.03263 (2019).

• Exactly what I was looking for, thanks! If I may ask a stupid question: what is the difference between the normalization constant and the partition function? Is it simply a question of terminology? – yawn Dec 27 '20 at 5:42
• My understanding is that the partition function is a particular case of a normalization constant (see en.wikipedia.org/wiki/Boltzmann_distribution). – Trisoloriansunscreen Dec 27 '20 at 6:21