0
$\begingroup$

In the Deep Learning book, when Goodfellow is trying to derive the MLE equation, he scales the following equation by $1/m$:

enter image description here

and then derives the following:

enter image description here

How does dividing $1/m$ is turned into the expected value?

$\endgroup$
1
  • 1
    $\begingroup$ The Law of Large Numbers. $\endgroup$ – stans Sep 18 '19 at 3:29
0
$\begingroup$

That's related to the Monte-Carlo method and law of large number.

Considering the coin toss problem, if you toss the same coin for 100 times and get heads 70 times and tails 30 times, then you would say the probability that a head appears is: $$ p(head) = \frac{1}{N_t}\sum_{N_t} O(head) = \frac{70}{100} = 70\% ,$$ where $N_t=100$ represents the total number you toss and $O(\cdot)$ represents the observation (head or tail in this case). Ideally, the approximation would be more accurate as $N_t \rightarrow \infty$.

Go back to your question, the main idea of maximum likelihood estimation is to find a parameter $\theta$ such that the probability of observations are maximized. In most cases, we can't directly access the true probability of x $p(x)$ but samples, that is, the observations $O(\cdot)$.

Now, start from your second equation, and by definition, $$E_{x \sim p_{data}(x)}\log p_{model}(x;\theta) = \sum p_{data(x)}\log p_{model}(x;\theta)$$ Again, we don't the true probability distribution $p_{data}(x)$, but we have samples. By Monte-Carlo method, we have $$\sum p_{data(x)}\log p_{model}(x;\theta) \approx \frac{1}{m}\sum_{i=1}^{m}\log p_{model}(x^{(i)};\theta)$$ Note that the approximation would be more accurate as $m \rightarrow \infty$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.