How to fit a unnormalized parametric distribution with MLE?

Question

I'm somewhat familiar with parametric estimation using MLE in the context of fitting the parameters of a distribution given a sample. Is there a way of generalizing this approach to unnormalized models (for instance, neural networks)? Naïvely maximizing the predicted log-likelihood would simply lead to the model predicting high values everywhere.

Answer 1

You are right that naive maximization of the likelihood would lead to blindly predicting high values.

You should look into Energy-based models (EBMs). EBMs define a likehood out of an arbitrary function $E_\theta(x)$ ("the energy function"): $$p_\theta(x)=\frac{\exp(−E_\theta (x))}{Z(\theta)}$$ where $x$ is an input, $\theta$ is the model's parameters (e.g., neural network weights) and $Z(\theta)$ is the partition function ($\int_x \exp(-E_\theta(x))$, which is not explicitly defined by the model.

The training of EBMs aims to choose a parameter set $\theta$ that would maximize $p_\theta(X)$ for the training distribution. This is achieved by using sampling methods to approximate $Z(\theta)$ during training, so we can train the model as if we had $Z(\theta)$ defined. The end result (ideally) is a model that maps the input $x$ to an output $E_\theta(x)$ which can be read out as an unnormalized negative-log-likelihood.

References:

https://en.wikipedia.org/wiki/Energy_based_model

LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., & Huang, F. (2006). A tutorial on energy-based learning. Predicting structured data, 1(0).

Grathwohl, Will, et al. "Your classifier is secretly an energy based model and you should treat it like one." arXiv preprint arXiv:1912.03263 (2019).