# Maximum Likelihood Estimation and expected value

In the Deep Learning book, when Goodfellow is trying to derive the MLE equation, he scales the following equation by $$1/m$$:

and then derives the following:

How does dividing $$1/m$$ is turned into the expected value?

• The Law of Large Numbers. Commented Sep 18, 2019 at 3:29
• stats.stackexchange.com/questions/434750/… this helped me a lot to figure it out Commented Sep 3, 2021 at 7:06
• The answer to this question is bassically the answer to this question. Commented Jan 6, 2023 at 16:16

Considering the coin toss problem, if you toss the same coin for 100 times and get heads 70 times and tails 30 times, then you would say the probability that a head appears is: $$p(head) = \frac{1}{N_t}\sum_{N_t} O(head) = \frac{70}{100} = 70\% ,$$ where $$N_t=100$$ represents the total number you toss and $$O(\cdot)$$ represents the observation (head or tail in this case). Ideally, the approximation would be more accurate as $$N_t \rightarrow \infty$$.
Go back to your question, the main idea of maximum likelihood estimation is to find a parameter $$\theta$$ such that the probability of observations are maximized. In most cases, we can't directly access the true probability of x $$p(x)$$ but samples, that is, the observations $$O(\cdot)$$.
Now, start from your second equation, and by definition, $$E_{x \sim p_{data}(x)}\log p_{model}(x;\theta) = \sum p_{data(x)}\log p_{model}(x;\theta)$$ Again, we don't the true probability distribution $$p_{data}(x)$$, but we have samples. By Monte-Carlo method, we have $$\sum p_{data(x)}\log p_{model}(x;\theta) \approx \frac{1}{m}\sum_{i=1}^{m}\log p_{model}(x^{(i)};\theta)$$ Note that the approximation would be more accurate as $$m \rightarrow \infty$$