I have a set of random variables $X_1$, $X_2$, $X_3$, ... $X_n$. They are continuous and $0 \leq X_i \leq 1$. And Let's assume they are i.i.d from the same distribution.

How do I evaluate the KL-divergence (or other distance metric defined over PDFs) between the distribution of these random variables to a uniform distribution? (if calculating the PDF of these random variables first is not an option).

  • $\begingroup$ Are $X_1$, $X_2$, etc... IID draws from the same underlying distribution? Or entirely arbitrary? For the former, you can compute an empirical pdf with enough data. For the later, I don't see what you can possibly say. $\endgroup$ – Matthew Gunn Sep 5 '16 at 2:19
  • $\begingroup$ @MatthewGunn I believe we have to assume that they are i.i.d. However, I don't have enough data to calculate the PDF. $\endgroup$ – Haohan Wang Sep 5 '16 at 3:29
  • 1
    $\begingroup$ I don't understand what you expect: you're trying to calculate KL divergence, a quantity that explicitly depends on the distribution of your random variable, without any information? Do you at least have numerical results for the approximate distribution? $\endgroup$ – Alex R. Sep 5 '16 at 18:35
  • $\begingroup$ @AlexR. I have the numerical results. The problem is that I want to estimate the KL divergence of this distribution over a uniform distribution, and I also want to make sure the solution is differentiable. But traditional methods to calculate the PDF first rely on sorting those random variables first, which is not differentiable. $\endgroup$ – Haohan Wang Sep 5 '16 at 18:49
  • $\begingroup$ So it sounds like you have a histogram for your random variables, but want it to be differentiable. In that case your only options would be some kind of smooth curve approximations, either through spline interpolation, mollifiers, etc. $\endgroup$ – Alex R. Sep 5 '16 at 19:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.