Maximum Likelihood Estimation and expected value

Question

In the Deep Learning book, when Goodfellow is trying to derive the MLE equation, he scales the following equation by $1/m$:

and then derives the following:

How does dividing $1/m$ is turned into the expected value?

Answer 1

Here you can observe that the expectation is with respect to $\mathbf{x} \sim \hat{p}_\text{data}$ which is the empirical distribution function of the sample. The empirical distribution is generated by taking random data points from the observed data vector via the algorithm:

$$\boldsymbol{x} = \boldsymbol{x}^{(i)} \quad \quad \quad \quad \quad i \sim \text{U} \{ 1,...,m \}.$$

Consequently, for any function $f$ that depends on $\boldsymbol{x}$ and $\boldsymbol{\theta}$ we have:

$$\begin{align} \mathbb{E}_{\mathbf{x} \sim \hat{p}_\text{data}}(f(\boldsymbol{x},\boldsymbol{\theta})) &= \mathbb{E}(f(\boldsymbol{x},\boldsymbol{\theta}) | \boldsymbol{x} \sim \hat{p}_\text{data}) \\[12pt] &= \mathbb{E}(f(\boldsymbol{x},\boldsymbol{\theta}) | \boldsymbol{x} = \boldsymbol{x}^{(i)}, i \sim \text{U} \{ 1,...,m \}) \\[12pt] &= \sum_{i=1}^m f(\boldsymbol{x}^{(i)},\boldsymbol{\theta}) \cdot \text{U}(i| 1,...,m) \\[6pt] &= \sum_{i=1}^m f(\boldsymbol{x}^{(i)},\boldsymbol{\theta}) \cdot \frac{1}{m} \\[6pt] &= \frac{1}{m} \sum_{i=1}^m f(\boldsymbol{x}^{(i)},\boldsymbol{\theta}). \\[6pt] \end{align}$$

Answer 2

That's related to the Monte-Carlo method and law of large number.

Considering the coin toss problem, if you toss the same coin for 100 times and get heads 70 times and tails 30 times, then you would say the probability that a head appears is: $$ p(head) = \frac{1}{N_t}\sum_{N_t} O(head) = \frac{70}{100} = 70\% ,$$ where $N_t=100$ represents the total number you toss and $O(\cdot)$ represents the observation (head or tail in this case). Ideally, the approximation would be more accurate as $N_t \rightarrow \infty$.

Go back to your question, the main idea of maximum likelihood estimation is to find a parameter $\theta$ such that the probability of observations are maximized. In most cases, we can't directly access the true probability of x $p(x)$ but samples, that is, the observations $O(\cdot)$.

Now, start from your second equation, and by definition, $$E_{x \sim p_{data}(x)}\log p_{model}(x;\theta) = \sum p_{data(x)}\log p_{model}(x;\theta)$$ Again, we don't the true probability distribution $p_{data}(x)$, but we have samples. By Monte-Carlo method, we have $$\sum p_{data(x)}\log p_{model}(x;\theta) \approx \frac{1}{m}\sum_{i=1}^{m}\log p_{model}(x^{(i)};\theta)$$ Note that the approximation would be more accurate as $m \rightarrow \infty$