Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
My stat prof basically said, if given one of the following three, you can find the other two:
But my econometrics professor said CDFs are more fundamental than PDFs because there are examples where you can have a CDF but the PDF isn't defined.
Are CDFs more fundamental than PDFs? How do I know whether a PDF or a MGF can be derived from a CDF?
Every probability distribution on (a subset of) $\mathbb R^n$ has a cumulative distribution function, and it uniquely defines the distribution. So, in this sense, the CDF is indeed as fundamental as the distribution itself.
A probability density function, however, exists only for (absolutely) continuous probability distributions. The simplest example of a distribution lacking a PDF is any discrete probability distribution, such as the distribution of a random variable that only takes integer values.
Of course, such discrete probability distributions can be characterized by a probability mass function instead, but there are also distributions that have neither and PDF or a PMF, such as any mixture of a continuous and a discrete distribution:
(Diagram shamelessly stolen from Glen_b's answer to a related question.)
There are even singular probability distributions, such as the Cantor distribution, which cannot be described even by a combination of a PDF and a PMF. Such distributions still have a well defined CDF, though. For example, here is the CDF of the Cantor distribution, also sometimes called the "Devil's staircase":
(Image from Wikimedia Commons by users Theon and Amirki, used under the CC-By-SA 3.0 license.)
The CDF, known as the Cantor function, is continuous but not absolutely continuous. In fact, it is constant everywhere except on a Cantor set of zero Lebesgue measure, but which still contains infinitely many points. Thus, the entire probability mass of the Cantor distribution is concentrated on this vanishingly small subset of the real number line, but every point in the set still individually has zero probability.
There are also probability distributions that do not have a moment-generating function. Probably the best known example is the Cauchy distribution, a fat-tailed distribution which has no well-defined moments of order 1 or higher (thus, in particular, having no well-defined mean or variance!).
All probability distributions on $\mathbb R^n$ do, however, have a (possibly complex-valued) characteristic function), whose definition differs from that of the MGF only by a multiplication with the imaginary unit. Thus, the characteristic function may be regarded as being as fundamental as the CDF.
I believe your econometrics professor was thinking something along the following lines.
Consider the function $F$ with domiain $[0, 1]$ defined by
$$F(x) = \frac{1}{2}x \ \text{for} \ x < \frac{1}{2} $$ $$F(x) = \frac{1}{2}x + \frac{1}{2} \ \text{for} \ x \geq \frac{1}{2} $$
This is a discontinuous function, but a completely valid CDF for some probability distribution on $[0, 1]$. Note that, using this distribution
$$ P\left(\left\{\frac{1}{2}\right\}\right) = \frac{1}{2} $$
There is no function $f$ that serves as a PDF for this distribution, even though there is a CDF.
It's easy enough to check this is true in this simple example if you've seen this kind of thing before. Suppose there is such a pdf $f$, we will show it must have an impossible property, and hence cannot exist.
By the definition of a PDF, we must have
$$ \int_0^x f(t) dt = F(x) - F(0) = \frac{1}{4}x $$
for all $0 < x < \frac{1}{2}$. A function that integrates to a linear function must be constant (technically constant almost everywhere), so we conclude that
$$ f(x) = \frac{1}{4} \ \text{for} \ x < \frac{1}{2} $$
In the same way, but integrating starting at one, moving towards zero, and ending at $x > \frac{1}{2}$, we reach the same conclusion
$$ f(x) = \frac{1}{4} \ \text{for} \ x > \frac{1}{2} $$
So we have determined $f$ everywhere except for $f\left(\frac{1}{2}\right)$. But it really does not matter what $f\left(\frac{1}{2}\right)$ is, it cannot have the integration property desired. Since
$$ P\left(\left\{\frac{1}{2}\right\}\right) = \frac{1}{2} $$
we would need
$$ \int_{\frac{1}{2} - \epsilon}^{\frac{1}{2} + \epsilon} f(t) dt > \frac{1}{2} $$
for every interval containing $\frac{1}{2}$. But in fact the value of any integral is unaffected by changing the value of a function at any single point, so
$$ \int_{\frac{1}{2} - \epsilon}^{\frac{1}{2} + \epsilon} f(t) dt = \int_{\frac{1}{2} - \epsilon}^{\frac{1}{2} + \epsilon} \frac{1}{4} dt = \frac{1}{2} \epsilon $$
So there's no way out, a function such as $f$ cannot exist.
You can recover the spirit of a PDF, but you must use more sophisticated mathematical objects, either a measure or a distribution.
Ilmari gives a good answer from a theoretical perspective. However, one may also ask what purposes the density (pdf) and the distribution function (pdf) serve for practical computations. This could clarify for which situations one is more directly useful than the other.
For probability distributions on $\mathbb{R}$ the distribution function directly gives probabilities of all intervals $(-\infty,x]$. From these probabilities, the probability of a finite union of intervals can be computed by elementary arithmetic. For all practical purposes I can think of, these are the only probabilities you would like to be able to compute. It may be theoretically convenient to express these $-$ or probabilities of more general sets $-$ as integrals, but for the actual computation we effectively need the distribution function.
The density is, however, essential for statistics, as the likelihood is defined in terms of the density. Thus if we want to compute the maximum likelihood estimate, we directly need the density.
If we turn to the comparison of an empirical and a theoretical distribution, both can be useful, but methods such as pp- and qq-plots based on the distribution function are often preferred.
For probability distributions on $\mathbb{R}^d$ for $d \geq 2$ the distribution function plays a less prominent role. One reason is that the probabilities for many sets of interest (balls, ellipsoids, cones etc.) cannot easily be computed from it.
