We all know the equation for the PDF of a Gaussian distribution, right?
$$ f_X(x\vert\mu,\sigma^2) = \dfrac{1}{\sqrt{2\pi\sigma^2}}\exp\bigg[{-\dfrac{1}{2}\bigg(\dfrac{x-\mu}{\sigma}\bigg)^2}\bigg] $$
However, this also is a valid equation for the PDF of a Gaussian distribution.
$$ g_X(x\vert\mu,\sigma^2) = \begin{cases} \dfrac{1}{\sqrt{2\pi\sigma^2}}\exp\bigg[{-\dfrac{1}{2}\bigg(\dfrac{x-\mu}{\sigma}\bigg)^2}\bigg], & x\ne 0 \\ 0, & x = 0 \end{cases} \ $$
The two differ at just that one point, $x=0$, meaning that their integrals are equal. These represent the same distribution.
The integral is the CDF.
Further, not every CDF has a corresponding PDF. The math winds up being a bit exotic, but it is possible to construct such a CDF. A standard example is the Cantor distribution.
So(As mentioned in the comments, there is a sense in which "almost all" CDFs arebehave this way and lack corresponding PDF.)
So for a random variable $X$, the CDF is defined uniquely and always, while PDFs arethe PDF is defined ambiguously and sometimes notif it is defined at all! This makes the CDF the natural place to operate. Imagine trying to do a test like Kolmogorov-Smirnov (KS) on my $f_X(x)$ and $g_X(x)$. For $\mu=0$ and $\sigma^2 = 0$, $f$ and $g$ would differ by a vertical distance of $0.399$ at $x=0$, which sounds like a lot, even though they correspond to the same distribution.