# Maximum likelihood as minimizing the dissimilarity between the empirical distriution and the model distribution

I am reading Ian Goodfellow "Deep Learning" book. At page 128 it says

One way to interpret maximum likelihood estimation is to view it as minimizing the dissimilarity between the empirical distribution $$\hat{p}_{\text{data}}$$, defined by the training set and the model distribution, with the degree of dissimilarity between the two measured by the KL divergence. The KL divergence is given by

$$D_{KL} (\hat{p}_{\text{data}} || p_{\text{model}}) = \mathbb{E}_{\mathbf{x} \sim \hat{p}_{\text{data}}} [\log \hat{p}_{\text{data}} - \log p_{\text{model}}(\mathbf{x})]$$

Starting from the definition of maximum likelihood estimator written in the text:

$$\mathbf{\theta}_{ML} = \arg\max_{\theta} p_{\text{model}}(\mathbb{X}; \mathbf{\theta})$$

Is there a formal proof for this? What is the intuition behind the formulation of maximum likelihood estimator as minimizing the KL divergence between the empirical distribution and the model distribution?

This is a late response but I hope it may help:

Proof (this proof is basically a summary of the explanations from the author):

To go for the proof, first we can follow the procedure given by the author which allows us to have a more convenient expression:

\begin{aligned} \theta_{ML} &=\arg \max_\theta p_{model}(\mathbb{X};\theta)\\ &= \arg\max_\theta\prod_{i=1}^m p_{model}(x^{(i)};\theta) \\ &= \arg\max_\theta\sum_{i=1}^m \log(p_{model}(x^{(i)};\theta)) \\ &= \arg\max_\theta \frac{1}{m}\sum_{i=1}^m \log(p_{model}(x^{(i)};\theta))\\ &= \arg\max_\theta \sum_{i=1}^m \frac{1}{m} \log(p_{model}(x^{(i)};\theta)) \end{aligned}

Note this last expression is the expectation of this $$\log(p_{model}(x;\theta))$$ function with respect to the empirical distribution defined by the training data ($$\hat{p}_{data}$$) which puts a probability of $$1/m$$ on each of the $$m$$ points $$x^{(1)},x^{(2)},...,x^{(m)}$$. So we can equivalently write this last expression as:

$$\theta_{ML} = \arg\max_{\theta}\mathbb{E}_{x\sim \hat{p}_{data}} \log(p_{model}(x;\theta))$$

Hence, obtaining the value of $$\theta$$ which satisfies this expresion will maximize the likelihood of $$p_{model}(x,\theta)$$ being the statistical model that best fits our set of data samples $$\mathbb{X}$$.

But we can also get the parameter $$\theta$$ that maximizes the likelihood using the KL divergence of the probability distributions $$\hat{p}_{data}$$ (empirical distribution of $$\mathbb{X}$$) and $$p_{model}$$ (our statistical model that we are using to fit $$\mathbb{X}$$):

$$D_{\text{KL}}(\hat{p}_{data} \parallel p_{model}) = \mathbb{E}_{x\sim \hat{p}_{data}} [ \log(\hat{p}_{data}(x)) - \log(p_{model}(x;\theta))]$$

This is because of the explanation given by the author, which is that $$\mathbb{E}_{x\sim \hat{p}_{data}} \log(\hat{p}_{data}(x))$$ does not depend on $$\theta$$ (it only depends on the data generating process), so it can be trated as a constant. Hence we can adress the same problem of finding the value of $$\theta$$ that maximizes the likelihood by minimizing this KL divergence, because this is the same as minimizing:

$$\mathbb{E}_{x\sim \hat{p}_{data}} [- \log(p_{model}(x;\theta))]$$

Which is just the negative form of the simplified expression for $$\theta_{ML}$$ that we have written earlier from the perspective of the likelihood.

So, to sum up, we can also calculate the parameter $$\theta_{ML}$$ by: $$\theta_{ML} = \arg\min_{\theta} D_{\text{KL}}(\hat{p}_{data} \parallel p_{model})= \arg\min_{\theta} \mathbb{E}_{x\sim \hat{p}_{data}} [- \log(p_{model}(x;\theta))]$$

Intuition:

With this said, I believe we can think of using the KL divergence for maximizing the likelihood as a way of making the predicted distribution $$(p_{model}(\mathbb{X},\theta))$$ as close as possible to the empirical distribution.

Thereby with our predicted distribution and by sampling it, we would be able to obtain a set of samples similar to the initial ones ($$\mathbb{X}$$). So this may mean that we have correctly calculate the true distribution of the data $$\mathbb{X}$$.

• There's a detailed discussion in the context of parameter survival models in this paper by Nils Lid Hjort: jstor.org/stable/1403683 The formal expression of idea goes back at least to Huber in 1967, but that paper is harder to find. Sep 9, 2020 at 22:13