Let $P(x)$ and $Q(x)$ be two pdfs. Let us say that $P(x)$ is the original baseline distribution and $Q(x)$ is the model (or estimate) distribution. I wanted to take the KL Divergence (or the 'distance') between the two distributions.
Which one should I perform: $KL(P,Q)$ or $KL(Q,P)$?
I am using the definition: $KL(p,q) = \displaystyle\int p(x) \ln \dfrac{p(x)}{q(x)}dx$
If $p(x)$ is the true distribution, and $q(x)$ is your model, then you want to be computing
$$KL(p,q) = \int{p(x)\ln{\frac{p(x)}{q(x)}}}dx.$$
Intuitively, you can see this as the information lost when using $q(x)$ as an approximation of $p(x)$. If you expand the log, it is easier to see that you are subtracting the expected log likelihood of $q(x)$ with respect to $p(x)$, from the true expected log likelihood of $p(x)$ leaving you with a non-negative value expressing said lost information.
When $KL(p,q) > 0$ then the model $q(x)$ is less likely to explain the data compared to the true distribution $p(x)$ (in expectation) and hence a indication of information loss.
When $KL(p,q) = 0$, then $q(x) = p(x)$ for all $x$ satisfying $q(x) > 0$ (converse holds as well). In this scenario, no information is lost.
Assuming that $p_i$ and $q_i$ are discrete, $n$-dimensional probability distributions. In Information Geometry it is proved that for two nearby probability distributions $$q_i=p_i+dp_i$$ with $\sum_i^n dp_i=0$, the infinitesimal distance $ds$ is $$ds \approx \sqrt{2 \; KL(p,q)}.$$ This expression immediately shows that the Kullback-Leibler divergence has the dimension of a length squared, $ds^2$, or an area.
For two probability distributions $p_i$ and $q_i$ which are not close to each other, the proper symmetric(!) geometrical distance is twice the Bhattacharyya angle, or $$s=2\, \arccos \sum_i^n \sqrt{p_i} \; \sqrt{q_i}.$$ This is the arc-length of a geodesic connecting $p_i$ and $q_i$ along a great circle on a $(n-1)$-dimensional hyper-sphere. The distance between two probability distributions is an angle!
For more details see our Kangaroo paper.
