In the comments of this question, it was pointed out that, when comparing two distributions, it is more natural and more general use the cumulative distribution (CDF) instead of the distribution (PDF).
The question is, why? I.e. what are the advantages (and/or disadvantages) of using the CDF instead of the PDF that make it more "more natural and general"?
We all know the equation for the PDF of a Gaussian distribution, right?
$$ f_X(x\vert\mu,\sigma^2) = \dfrac{1}{\sqrt{2\pi\sigma^2}}\exp\bigg[{-\dfrac{1}{2}\bigg(\dfrac{x-\mu}{\sigma}\bigg)^2}\bigg] $$
However, this also is a valid equation for the PDF of a Gaussian distribution.
$$ g_X(x\vert\mu,\sigma^2) = \begin{cases} \dfrac{1}{\sqrt{2\pi\sigma^2}}\exp\bigg[{-\dfrac{1}{2}\bigg(\dfrac{x-\mu}{\sigma}\bigg)^2}\bigg], & x\ne 0 \\ 0, & x = 0 \end{cases} \ $$
The two differ at just that one point, $x=0$, meaning that their integrals are equal. These represent the same distribution.
The integral is the CDF.
Further, not every CDF has a corresponding PDF. The math winds up being a bit exotic, but it is possible to construct such a CDF. A standard example is the Cantor distribution.
(As mentioned in the comments, there is a sense in which "almost all" CDFs behave this way and lack corresponding PDF.)
So for a random variable $X$, the CDF is defined uniquely and always, while the PDF is defined ambiguously if it is defined at all! This makes the CDF the natural place to operate. Imagine trying to do a test like Kolmogorov-Smirnov (KS) on my $f_X(x)$ and $g_X(x)$. For $\mu=0$ and $\sigma^2 = 0$, $f$ and $g$ would differ by a vertical distance of $0.399$ at $x=0$, which sounds like a lot, even though they correspond to the same distribution.
Others have already pointed out that a PDF can be modified arbitrarily on a set of probability measure zero, and this still gives a valid PDF for the random variable. That is true, but it is somewhat of an artificial answer to your question. In such cases, for almost all random variables we deal with in practice, there is still a "natural" PDF (e.g., the continuous one) which is the one we usually use. These could in theory be compared as a means of looking at the "distance" between random variables, so your question is a reasonable one.
So, let's restrict attention to cases where we are comparing random variables that have a continuous PDF, and let's take the continuous PDF as the "natural" version used for the comparison. Even with this restriction, it is still possible to create a "spike" in a PDF which can be arbitrarily large without having a substantial effect on the CDF. To see this, consider a random variable $X$ and suppose we form the mixture random variable:
$$Y \equiv IG + (1-I)X \quad \quad \quad \quad \quad G \sim \text{N}(x_0, \epsilon^2) \quad \quad \quad \quad \quad I\sim \text{Bern}(\epsilon).$$
The effect of using this mixture distribution when $\epsilon$ is small is that the distribution of $Y$ looks roughly proportionate to the distribution of $X$, except that it has a continuous "spike" concentrated around the value $x_0$. Taking $\epsilon \rightarrow 0$ there is convergence in probability $Y \rightarrow X$ and convergence in the corresponding distributions (i.e., $F_Y \rightarrow F_X$). However, when we take this limit the "spike" in the PDF of $Y$ gets arbitrarily large, so the distance between the PDFs at the point $x_0$ diverges to infinity.
So, in terms of distance between $X$ and $Y$, if $\epsilon$ is small, what do you do here? Intuitively, the random variables are very close to each other (as $\epsilon \rightarrow 0$ they converge) so the "distance" should be small. However, if we are judging things from the PDFs, we see that there is a point where these PDFs are growing apart infinitely, so perhaps this means that the "distance" ought to be large?
This example indicates that if you are using the PDFs to determine the "distance" between random variables, you are going to have to find a way to resolve this type of case. If you think there is little distance between these random variables then the distance measure constructed on the PDFs should reflect this, which would mean the distance remains small even if the PDFS grow apart infinitely at a point (or indeed, at any countable number of points). I will leave it to you to think about how you might construct such a measure.
