In the Deep Learning book, when Goodfellow is trying to derive the MLE equation, he scales the following equation by $1/m$:

enter image description here

and then derives the following:

enter image description here

How does dividing $1/m$ is turned into the expected value?

  • 1
    $\begingroup$ The Law of Large Numbers. $\endgroup$
    – stans
    Commented Sep 18, 2019 at 3:29
  • $\begingroup$ stats.stackexchange.com/questions/434750/… this helped me a lot to figure it out $\endgroup$
    Commented Sep 3, 2021 at 7:06
  • $\begingroup$ The answer to this question is bassically the answer to this question. $\endgroup$
    – neoglez
    Commented Jan 6, 2023 at 16:16

1 Answer 1


That's related to the Monte-Carlo method and law of large number.

Considering the coin toss problem, if you toss the same coin for 100 times and get heads 70 times and tails 30 times, then you would say the probability that a head appears is: $$ p(head) = \frac{1}{N_t}\sum_{N_t} O(head) = \frac{70}{100} = 70\% ,$$ where $N_t=100$ represents the total number you toss and $O(\cdot)$ represents the observation (head or tail in this case). Ideally, the approximation would be more accurate as $N_t \rightarrow \infty$.

Go back to your question, the main idea of maximum likelihood estimation is to find a parameter $\theta$ such that the probability of observations are maximized. In most cases, we can't directly access the true probability of x $p(x)$ but samples, that is, the observations $O(\cdot)$.

Now, start from your second equation, and by definition, $$E_{x \sim p_{data}(x)}\log p_{model}(x;\theta) = \sum p_{data(x)}\log p_{model}(x;\theta)$$ Again, we don't the true probability distribution $p_{data}(x)$, but we have samples. By Monte-Carlo method, we have $$\sum p_{data(x)}\log p_{model}(x;\theta) \approx \frac{1}{m}\sum_{i=1}^{m}\log p_{model}(x^{(i)};\theta)$$ Note that the approximation would be more accurate as $m \rightarrow \infty$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.