Maximum Likelihood Estimation and expected value

In the Deep Learning book, when Goodfellow is trying to derive the MLE equation, he scales the following equation by $$1/m$$:

and then derives the following:

How does dividing $$1/m$$ is turned into the expected value?

• The Law of Large Numbers. Sep 18, 2019 at 3:29
• stats.stackexchange.com/questions/434750/… this helped me a lot to figure it out Sep 3, 2021 at 7:06
• The answer to this question is bassically the answer to this question. Jan 6, 2023 at 16:16

That's related to the Monte-Carlo method and law of large number.

Considering the coin toss problem, if you toss the same coin for 100 times and get heads 70 times and tails 30 times, then you would say the probability that a head appears is: $$p(head) = \frac{1}{N_t}\sum_{N_t} O(head) = \frac{70}{100} = 70\% ,$$ where $$N_t=100$$ represents the total number you toss and $$O(\cdot)$$ represents the observation (head or tail in this case). Ideally, the approximation would be more accurate as $$N_t \rightarrow \infty$$.

Go back to your question, the main idea of maximum likelihood estimation is to find a parameter $$\theta$$ such that the probability of observations are maximized. In most cases, we can't directly access the true probability of x $$p(x)$$ but samples, that is, the observations $$O(\cdot)$$.

Now, start from your second equation, and by definition, $$E_{x \sim p_{data}(x)}\log p_{model}(x;\theta) = \sum p_{data(x)}\log p_{model}(x;\theta)$$ Again, we don't the true probability distribution $$p_{data}(x)$$, but we have samples. By Monte-Carlo method, we have $$\sum p_{data(x)}\log p_{model}(x;\theta) \approx \frac{1}{m}\sum_{i=1}^{m}\log p_{model}(x^{(i)};\theta)$$ Note that the approximation would be more accurate as $$m \rightarrow \infty$$