I am watching this great lecture by Nando De Freitas.

He establishes the KL divergence by using maximum liklihood estimation.

However, there is one step I don't really understand.

I do understand the steps from a math standpoint. I just wonder why he wants to measure the similarity between the distributions P(x|theta) and P(x|theta_0).

I also wonder how I can imagine the distribution P(x|theta_0).

As I understand, theta_0 is just the parameter of the bias term.

Why do we even need a distribution for this?

Thanks!