From what I've read the KL-Divergence between $P||Q$ is the extra amount of "bits" you need to describe $P$ if you are encoding it with $Q$.(Analysis of Kullback-Leibler divergence).
I want to know when does $Q$ gives me at least "some" information about $P$?
My logic is as follows:
- We normally need $H(P)$ bits to describe $P$.
- If I am given $Q$ then I would only need $KL(P||Q)$ more bits to specify $P$.
- Therefore as long as $KL(P||Q) < H(P)$, I have been able to reduce the amount of bits needed to describe $P$.
Can I conclude that if $KL(P||Q) < H(P)$ then $Q$ has predictive power ( > 0) over $P$?
Additionally, if $KL(P||Q) > H(P)$ then there is no predictive power.