2
$\begingroup$

In the online version of the Deep Learning book on chapter 5 the estimator for likelihood function is defined as:

enter image description here

That is the product of individual probabilities. After taking the log it arrives at the log-likelihood funciton (Eq.1):

enter image description here

It then rescales the above by dividing it by m to obtain a version expressed as the expected value (Eq.2):

enter image description here

OK. Here is my first question: The expected value is defined as enter image description here

And so I think the expected value of log will be the below expression which is not the same as Eq.1 in the book.

enter image description here

The only way for this to be true(that is to divide the Eq.1 by m and arrive at Eq.2 as claimed by the book author) is for the probabilities of p(data) to be uniform. But this is also not a valid assumption. So I don't see what I am missing here?


Next, the book argues that maximizing the above log-likelihood function (Eq.2) is same as minimizing the KL divergence:

enter image description here

Or more simply just minimizing the second term. And so the author says that either way we arrive at the same function as Eq.2.

enter image description here

On the other hand, from the Wikipedia page the cross entropy of two probability is defined as :

enter image description here

I can understand this definition as the expected value of log(q) which is same as the expression in the Eq.2. but not Eq.1

From the same Wiki page the likelihood definition is given as below which is different than the likelihood function definition from the book(above). Here the probability of q (model) has been raised to the number of occurrences; which then on taking log it is understandable to see it as the expected value.

enter image description here

So I am confused that first of all which definition of likelihood function is the correct one? Given the definition from the Wikipedia I can understand that maximizing the log-likelihood function is same as minimizing the cross-entropy function. However I cannot arrive at the same conclusion from the definition of the likelihood and the log-likelihood function given in the book, for the reasons I explained above.

$\endgroup$

2 Answers 2

3
$\begingroup$

Actually, the equation you wrote should use the empirical distribution: $$\mathbb E_{\mathbf x\sim \hat p_{data}}[\log p_{\text {model}}(\mathbf x;\theta)]=\sum_{i=1}^m \hat p_{data}(\mathbf x_i)\log p_{\text {model}}(\mathbf x_i;\theta)$$

And, since the empirical distribution is obtained using the data which is sampled iid, it'll be $1/m$. We typically expect empirical distribution to represent the true distribution as sample size increases, so even in your version of the equation, an approximate sign , i.e. $\approx$, wouldn't be illogical.

For your second question about the likelihood definitions, abbreviate $p_{\text{model}}(\mathbf x_i;\theta)$ as $p_i$, then the expressions are actually equivalent. Because, if a sample $\mathbf x_i$ appears $N_i$ times in the dataset (say the indices are $x_{i_1},...,x_{i_{N_i}}$, i.e. $x_{i_j}=x_{i_k}$), then in the first version of the likelihood you'll have the term $$L_i=\prod_{j=1}^{N_i} x_{i_j}$$, and in the second version of the likelihood, you'll have the term $$L_i=x_i^{N_i}$$ which are practically the same because they're multiplications of the same term $N_i$ times.

$\endgroup$
6
  • $\begingroup$ but for example, in linear regression, isn't it that we assume a normal distribution for our data, rather than being uniformly distributed(1/m)? $\endgroup$
    – ali
    Commented Dec 19, 2020 at 19:20
  • $\begingroup$ The true distribution can be normal, and the empirical distribution will resemble normal. Then, you'll get more samples around some parts of the distribution (e.g. mean), and even if you put $1/m$ probability mass to each of the samples, if several samples are the same, the effective mass for that sample will be larger. $\endgroup$
    – gunes
    Commented Dec 19, 2020 at 20:38
  • $\begingroup$ Do you mean it won't be wrong to approximately assume the samples to have uniform distribution(1/m) even though the true distribution is not, because 1/m is almost true for samples near the mean? This is the bit I struggle to understand. Would you mind elaborating it in your answer? $\endgroup$
    – ali
    Commented Dec 19, 2020 at 20:52
  • 3
    $\begingroup$ No, let's say your data is [1,1,1,1,2,2,3,3,3,4], and therefore m = 10 and each of these samples will have 1/10 probability mass. So, $p(x_1)=1/10, p(x_2)=1/10$ etc. But, the effective mass for the value $1$ is not $1/10$., it's $4/10$. When you sample a distribution, samples around higher probability regions will appear more in the data, so even if you assign $1/m$ to each sample $x_i$, the distinct values will have probabilities resembling the true distribution. $\endgroup$
    – gunes
    Commented Dec 19, 2020 at 20:58
  • 1
    $\begingroup$ aaaahaa! that was perfect ! thanks. it makes sense now :) $\endgroup$
    – ali
    Commented Dec 19, 2020 at 21:27
2
$\begingroup$

For your first question, the expression $$\frac{1}{m} \sum_{i=1}^m \log p_\text{model}(x_i;\theta)\tag{1}$$ is an average of iid random variables with expectation $$\mathbb E^X[\log p_\text{model}(X;\theta)]=\sum_{x} \log p_\text{model}\,(X;\theta) p_\text{data}(x)\tag{2}$$ whatever the distribution $p_\text{data}(\cdot)$ of $X$. Hence correct asymptotically in $m$.

For the second question, maximising (1) is asymptotically like maximising (2), i.e., like minimising $$-\mathbb E^X[\log p_\text{model}(X;\theta)]\tag{3}$$ yet again like minimising $$\mathbb E^X[\log p_\text{data}(X)]-\mathbb E^X[\log p_\text{model}(X;\theta)]\tag{4}$$

$\endgroup$
3
  • $\begingroup$ So does it mean that iid assumes the samples to have uniform distribution (1/m)? $\endgroup$
    – ali
    Commented Dec 19, 2020 at 20:54
  • $\begingroup$ And so in the definition of cross entropy (between the two definition p(model) and p(data) ), p(data) always is 1/m ? $\endgroup$
    – ali
    Commented Dec 19, 2020 at 20:59
  • 1
    $\begingroup$ Not at all, the $1/m$ is related to the number of observations in the iid sample, not to the number of possible values taken by any of the $X_i$'s $\endgroup$
    – Xi'an
    Commented Dec 19, 2020 at 21:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.