I am reading Ian Goodfellow "Deep Learning" book. At page 128 it says

One way to interpret maximum likelihood estimation is to view it as minimizing the dissimilarity between the empirical distribution $\hat{p}_{\text{data}}$, defined by the training set and the model distribution, with the degree of dissimilarity between the two measured by the KL divergence. The KL divergence is given by

$$ D_{KL} (\hat{p}_{\text{data}} || p_{\text{model}}) = \mathbb{E}_{\mathbf{x} \sim \hat{p}_{\text{data}}} [\log \hat{p}_{\text{data}} - \log p_{\text{model}}(\mathbf{x})]$$

Starting from the definition of maximum likelihood estimator written in the text:

$$\mathbf{\theta}_{ML} = \arg\max_{\theta} p_{\text{model}}(\mathbb{X}; \mathbf{\theta}) $$

Is there a formal proof for this? What is the intuition behind the formulation of maximum likelihood estimator as minimizing the KL divergence between the empirical distribution and the model distribution?


1 Answer 1


This is a late response but I hope it may help:

Proof (this proof is basically a summary of the explanations from the author):

To go for the proof, first we can follow the procedure given by the author which allows us to have a more convenient expression:

$$\begin{aligned} \theta_{ML} &=\arg \max_\theta p_{model}(\mathbb{X};\theta)\\ &= \arg\max_\theta\prod_{i=1}^m p_{model}(x^{(i)};\theta) \\ &= \arg\max_\theta\sum_{i=1}^m \log(p_{model}(x^{(i)};\theta)) \\ &= \arg\max_\theta \frac{1}{m}\sum_{i=1}^m \log(p_{model}(x^{(i)};\theta))\\ &= \arg\max_\theta \sum_{i=1}^m \frac{1}{m} \log(p_{model}(x^{(i)};\theta)) \end{aligned}$$

Note this last expression is the expectation of this $\log(p_{model}(x;\theta))$ function with respect to the empirical distribution defined by the training data ($\hat{p}_{data}$) which puts a probability of $1/m$ on each of the $m$ points $x^{(1)},x^{(2)},...,x^{(m)}$. So we can equivalently write this last expression as:

$$ \theta_{ML} = \arg\max_{\theta}\mathbb{E}_{x\sim \hat{p}_{data}} \log(p_{model}(x;\theta))$$

Hence, obtaining the value of $\theta$ which satisfies this expresion will maximize the likelihood of $p_{model}(x,\theta)$ being the statistical model that best fits our set of data samples $\mathbb{X}$.

But we can also get the parameter $\theta$ that maximizes the likelihood using the KL divergence of the probability distributions $\hat{p}_{data}$ (empirical distribution of $\mathbb{X}$) and $p_{model}$ (our statistical model that we are using to fit $\mathbb{X}$):

$$ D_{\text{KL}}(\hat{p}_{data} \parallel p_{model}) = \mathbb{E}_{x\sim \hat{p}_{data}} [ \log(\hat{p}_{data}(x)) - \log(p_{model}(x;\theta))]$$

This is because of the explanation given by the author, which is that $\mathbb{E}_{x\sim \hat{p}_{data}} \log(\hat{p}_{data}(x))$ does not depend on $\theta$ (it only depends on the data generating process), so it can be trated as a constant. Hence we can adress the same problem of finding the value of $\theta$ that maximizes the likelihood by minimizing this KL divergence, because this is the same as minimizing:

$$ \mathbb{E}_{x\sim \hat{p}_{data}} [- \log(p_{model}(x;\theta))]$$

Which is just the negative form of the simplified expression for $\theta_{ML}$ that we have written earlier from the perspective of the likelihood.

So, to sum up, we can also calculate the parameter $\theta_{ML}$ by: $$ \theta_{ML} = \arg\min_{\theta} D_{\text{KL}}(\hat{p}_{data} \parallel p_{model})= \arg\min_{\theta} \mathbb{E}_{x\sim \hat{p}_{data}} [- \log(p_{model}(x;\theta))] $$


With this said, I believe we can think of using the KL divergence for maximizing the likelihood as a way of making the predicted distribution $(p_{model}(\mathbb{X},\theta))$ as close as possible to the empirical distribution.

Thereby with our predicted distribution and by sampling it, we would be able to obtain a set of samples similar to the initial ones ($\mathbb{X}$). So this may mean that we have correctly calculate the true distribution of the data $\mathbb{X}$.

  • 3
    $\begingroup$ There's a detailed discussion in the context of parameter survival models in this paper by Nils Lid Hjort: jstor.org/stable/1403683 The formal expression of idea goes back at least to Huber in 1967, but that paper is harder to find. $\endgroup$ Sep 9, 2020 at 22:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.