Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I know KL divergence tries to measure how different 2 probability distributions are.
I know high correlation values between 2 sets of variables imply they are highly dependent on each other.
Will the probability distributions associated with both sets of variables have low KL divergence between them, i.e.: will they be similar?
Not necessarily, e.g. $Y=cX$ and $X\sim N(0,1),\ c>0$, which means $Y\sim N(0,c^2)$. The KL divergence between two univariate normals can be calculated as laid out in here, and yields: $$KL(p_x||p_y)=2\log c+\frac{1}{2c^2}-{1\over2}$$
This can be arbitrarily large as $c$ changes but the correlation is always $1$.
