How to fit a unnormalized parametric distribution with MLE? I'm somewhat familiar with parametric estimation using MLE in the context of fitting the parameters of a distribution given a sample. Is there a way of generalizing this approach to unnormalized models (for instance, neural networks)? Naïvely maximizing the predicted log-likelihood would simply lead to the model predicting high values everywhere.
 A: You are right that naive maximization of the likelihood would lead to blindly predicting high values.
You should look into Energy-based models (EBMs). EBMs define a likehood out of an arbitrary function $E_\theta(x)$ ("the energy function"):
$$p_\theta(x)=\frac{\exp(−E_\theta (x))}{Z(\theta)}$$
where $x$ is an input, $\theta$ is the model's parameters (e.g., neural network weights) and $Z(\theta)$ is the partition function ($\int_x \exp(-E_\theta(x))$, which is not explicitly defined by the model.
The training of EBMs aims to choose a parameter set $\theta$ that would maximize $p_\theta(X)$ for the training distribution. This is achieved by using sampling methods to approximate $Z(\theta)$ during training, so we can train the model as if we had $Z(\theta)$ defined. The end result (ideally) is a model that maps the input $x$ to an output $E_\theta(x)$ which can be read out as an unnormalized negative-log-likelihood.
References:
https://en.wikipedia.org/wiki/Energy_based_model

LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., & Huang, F. (2006). A tutorial on energy-based learning. Predicting structured data, 1(0).


Grathwohl, Will, et al. "Your classifier is secretly an energy based model and you should treat it like one." arXiv preprint arXiv:1912.03263 (2019).

