1
$\begingroup$

From what I've read the KL-Divergence between $P||Q$ is the extra amount of "bits" you need to describe $P$ if you are encoding it with $Q$.(Analysis of Kullback-Leibler divergence).

I want to know when does $Q$ gives me at least "some" information about $P$?

My logic is as follows:

  • We normally need $H(P)$ bits to describe $P$.
  • If I am given $Q$ then I would only need $KL(P||Q)$ more bits to specify $P$.
  • Therefore as long as $KL(P||Q) < H(P)$, I have been able to reduce the amount of bits needed to describe $P$.

Can I conclude that if $KL(P||Q) < H(P)$ then $Q$ has predictive power ( > 0) over $P$?

Additionally, if $KL(P||Q) > H(P)$ then there is no predictive power.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ H(P) is the average number of bits needed to describe a value randomly drawn from P, if you use a code based on perfect knowledge of P. Your train of thought seems rooted in the idea that the code is built with no knowledge of P. You can't gain information about P by approximating it with Q. $\endgroup$ – Jonny Lomond Feb 1 '18 at 22:40
  • $\begingroup$ I see, yes I was incorrectly assuming code was built w/o knowledge of P. Regarding your last statement, isn't mutual information (similar to KL-divergence but you need to bin the data) used for quantifying such things? Can KL-div help us do such things? $\endgroup$ – RM- Feb 2 '18 at 7:15
  • $\begingroup$ What do you mean by "predictive power" in this context? $\endgroup$ – DiveIntoML Feb 2 '18 at 16:09
  • $\begingroup$ I guess I am referring to a "goodness" of fit, if that makes sense. I basically want to compare two distributions and say how good is my model for predicting the truth but in normalized way. Normalized mutual information could be a valid option but I would have to "bin" the data, which I'd rather not. I was wondering if I could use KL-divergence and have a simple thumb rule that would say "pass" or "fail". $\endgroup$ – RM- Feb 2 '18 at 18:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.