5
$\begingroup$

I am watching this great lecture by Nando De Freitas.

He establishes the KL divergence by using maximum liklihood estimation.

However, there is one step I don't really understand. enter image description here

I do understand the steps from a math standpoint. I just wonder why he wants to measure the similarity between the distributions P(x|theta) and P(x|theta_0).

I also wonder how I can imagine the distribution P(x|theta_0).

As I understand, theta_0 is just the parameter of the bias term.

Why do we even need a distribution for this?

Improve this question
$\endgroup$
1
  • 3
    $\begingroup$ P(x|theta_0) is the idealized true distribution where the data came from. It is what you are trying to find as end goal by minimizing the distance between what you propose and the truth. It has nothing to do with bias. $\endgroup$
    – Cagdas Ozgenc
    Commented Nov 19, 2017 at 0:01

1 Answer 1

Reset to default
11
$\begingroup$

I just wonder why he wants to measure the similarity between the distributions $p(x|\theta)$ and $p(x|\theta_0)$.

You're kind of asking the wrong question. If we're in a setting where we're using MLE, then the idea behind it is that we're estimating the parameters of our model with the parameters that maximize the likelihood. It will probably be the case that the true likelihood ($p(x|\theta_0)$) isn't actually going to be in the parametric family of likelihoods we're working with!

What he's doing here is showing $that\ performing\ MLE\ is\ equivalent$ to minimizing the KL divergence between the true likelihood and the family of likelihoods we're using for the MLE. So while the true $p(x|\theta_0)$ might not actually be inside of the family of likelihoods you're performing MLE over, what this tells us is that the MLE from our family of likelihoods will be the closest in this family to the true distribution $in\ KL\ divergence$. This is nice because even if we're off in our model specification, we find that the MLE will still be close to the truth in some sense.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ so for p(x|θ_0), θ_0 is not the parameter of the bias term but the "optimal" parameters for which the likelihood would be highest? $\endgroup$
    – cmplx96
    Commented Nov 19, 2017 at 11:14
  • 1
    $\begingroup$ It's not the parameters for which the likelihood would be highest, but rather the true parameters for the likelihood (i.e. the parameters that actually generated the data). This is an important distinction, and at the core of frequentist inference (the idea that there actually is some true parameter $\theta_0$). Because you're working with finite data, it may be the case that the parameters that maximize the likelihood actually have a higher likelihood that the true parameters for the data that you see. $\endgroup$
    – aleshing
    Commented Nov 19, 2017 at 19:19
  • 1
    $\begingroup$ Thank you for the answer. I've included insights from this answer into Wikipedia, feel free to expand as you see fit: en.wikipedia.org/wiki/… $\endgroup$
    – Tal Galili
    Commented Jan 16, 2020 at 13:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.