Let $\mathbf{X}$ be a continuous random vector, $\mathbf{d}$ a sample of size $m$, and $\mathbb{P}_{\mathbf{X}|\mathbf{\theta}}$ a parametric model for the distribution of $\mathbf{X}$.

We can write the likelihood of the sample under this parametric model as:

$f(\mathbf{d}|\mathbf{\theta}) = \exp\left[-mH_\lambda(\hat{\mathbb{P}}_m||\mathbb{P}_{\mathbf{X}|\mathbf{\theta}})\right]$

where $\hat{\mathbb{P}}_m$ is the empirical distribution and $H_\lambda(\hat{\mathbb{P}}_m||\mathbb{P}_{\mathbf{X}|\mathbf{\theta}})$ is the cross-entropy between $\hat{\mathbb{P}}_m$ and $\mathbb{P}_{\mathbf{X}|\mathbf{\theta}}$ relatively to the Lebesgue measure.

Now, in the infinite sample limit, we have that $\hat{\mathbb{P}}_m \xrightarrow[m \to +\infty]{} \tilde{\mathbb{P}}_{\mathbf{X}}$ where $\tilde{\mathbb{P}}_{\mathbf{X}}$ is the unknown distribution of $\mathbf{X}$ that generated the sample.

Using the fact that $H_\lambda(\tilde{\mathbb{P}}_{\mathbf{X}}||\mathbb{P}_{\mathbf{X}|\mathbf{\theta}}) = H_\lambda(\tilde{\mathbb{P}}_{\mathbf{X}}) + D_{KL}(\tilde{P}_{\mathbf{X}}||\mathbb{P}_{\mathbf{X}|\mathbf{\theta}})$, in this limit, maximizing the likelihood is minimizing the KL divergence between the distribution of $\mathbf{X}$ and the parametric model.

As it has been asked in this thread (Maximum likelihood and Minimizing Kullback–Leibler divergence to the ecdf?! (i.e.: finite sample statement?)), is it possible to have the same statement but in the finite sample case ? I think I agree with this answer but it has been pointed in two other threads ((Maximum likelihood as minimizing the dissimilarity between the empirical distriution and the model distribution) and (Kullback–Leibler divergence when one measure is a sum of diracs)), that the section 5.5 of Deep Learning, Ian Goodfellow (https://www.deeplearningbook.org/contents/ml.html) states that it is possible.

I would like to use the formula that links the cross-entropy and the KL divergence but I guess it is not well defined when we use the empirical distribution in place of the distribution of $\mathbf{X}$.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.