In the online version of the Deep Learning book on chapter 5 the estimator for likelihood function is defined as:
That is the product of individual probabilities. After taking the log it arrives at the log-likelihood funciton (Eq.1):
It then rescales the above by dividing it by m to obtain a version expressed as the expected value (Eq.2):
OK. Here is my first question:
The expected value is defined as
And so I think the expected value of log will be the below expression which is not the same as Eq.1 in the book.
The only way for this to be true(that is to divide the Eq.1 by m and arrive at Eq.2 as claimed by the book author) is for the probabilities of p(data) to be uniform. But this is also not a valid assumption. So I don't see what I am missing here?
Next, the book argues that maximizing the above log-likelihood function (Eq.2) is same as minimizing the KL divergence:
Or more simply just minimizing the second term. And so the author says that either way we arrive at the same function as Eq.2.
On the other hand, from the Wikipedia page the cross entropy of two probability is defined as :
I can understand this definition as the expected value of log(q) which is same as the expression in the Eq.2. but not Eq.1
From the same Wiki page the likelihood definition is given as below which is different than the likelihood function definition from the book(above). Here the probability of q (model) has been raised to the number of occurrences; which then on taking log it is understandable to see it as the expected value.
So I am confused that first of all which definition of likelihood function is the correct one? Given the definition from the Wikipedia I can understand that maximizing the log-likelihood function is same as minimizing the cross-entropy function. However I cannot arrive at the same conclusion from the definition of the likelihood and the log-likelihood function given in the book, for the reasons I explained above.