MLE as an expectation over the empirical distribution

Question

I am reading Ian Goodfellow "Deep Learning" book. At page 128, it writes the maximum log-likelihood estimator and then says it is equivalent to the expectation over the empirical distribution

To obtain a more convenient but equivalent optimization problem, we observe that taking the logarithm of the likelihood does not change its arg max but does conveniently transform a product into a sum

$$\mathbf{\theta}_{ML} = \arg\max_{\theta} \sum\limits_{i=1}^{m} \log p_{\text{model}}(\mathbf{x}^{(i)}; \mathbf{\theta}) $$

Because the arg max does not change when we rescale the cost function, we can divide by $m$ to obtain a version of the criterion that is expressed as an expectation with respect to the empirical distribution $\hat{p}_{\text{data}}$ defined by the training data:

$$\mathbf{\theta}_{ML} = \arg\max_{\theta} \mathbb{E}_{\mathbf{x} \sim \hat{p}_{\text{data}}} \log p_{\text{model}} (\mathbf{x}; \mathbf{\theta}) $$

Can you provide a little insight / a proof for that? I have read on Wikipedia that the "empirical distribution" is a cumulative distribution function and

This cumulative distribution function is a step function that jumps up by 1/n at each of the n data points. Its value at any specified value of the measured variable is the fraction of observations of the measured variable that are less than or equal to the specified value.

Is this the definition of "empirical distribution" used by the book? Or rather it uses the probability density instead of the cumulative distribution?

How does it fit in the definition of log-likelihood as expectation over the empirical distribution, and why this definition follows from the definition of the first formula?

Answer 1

When you generate a random variable from the empirical distribution of your data, this essentially just means that you are "resampling" a random data point uniformly from your observed sample. The simplest way to frame this is in terms of a uniformly distributed random index number determining the data point that you are resampling. So if you have the data $(\mathbf{x}^{(1)},...,\mathbf{x}^{(m)})$ then you can get a random variable from the empirical distribution of the sample by taking:

$$S \sim \text{U} \{ 1,...,m \} \quad \quad \quad \mathbf{X} \equiv \mathbf{x}^{(S)} \sim \hat{p}_\text{data}.$$

Now, since $\mathbf{X}$ is a discrete random variable with support over the data vector, applying the definition of expectation gives:

$$\begin{equation} \begin{aligned} \mathbb{E}_{\mathbf{X}} ( \log p_\text{model} (\mathbf{X}; \theta) ) &= \sum_{i=1}^m \log p_\text{model} (\mathbf{x}^{(i)}; \theta) \cdot \mathbb{P}(\mathbf{X} = \mathbf{x}^{(i)} ) \\[6pt] &= \sum_{i=1}^m \log p_\text{model} (\mathbf{x}^{(i)}; \theta) \cdot \mathbb{P}(\mathbf{x}^{(S)} = \mathbf{x}^{(i)} ) \\[6pt] &= \sum_{i=1}^m \log p_\text{model} (\mathbf{x}^{(i)}; \theta) \cdot \mathbb{P}(S = i) \\[6pt] &= \sum_{i=1}^m \log p_\text{model} (\mathbf{x}^{(i)}; \theta) \cdot \frac{1}{m} \\[6pt] &= \frac{1}{m} \sum_{i=1}^m \log p_\text{model} (\mathbf{x}^{(i)}; \theta). \\[6pt] \end{aligned} \end{equation}$$

Hence, we have:

$$\begin{equation} \begin{aligned} \theta_{ML} &= \text{arg} \max_{\theta} \mathbb{E}_{\mathbf{X}} ( \log p_\text{model} (\mathbf{X}; \theta) ) \\[6pt] &= \text{arg} \max_{\theta} \frac{1}{m} \sum_{i=1}^m \log p_\text{model} (\mathbf{x}^{(i)}; \theta) \\[6pt] &= \text{arg} \max_{\theta} \sum_{i=1}^m \log p_\text{model} (\mathbf{x}^{(i)}; \theta). \\[6pt] \end{aligned} \end{equation}$$

This confirms the form asserted in your book. I am not really sure how this framing of the maximum likelihood estimator is useful (since there is no particular difficulty with the initial optimisation problem), but perhaps this author is able to squeeze some useful value out of it.