42
$\begingroup$

My stat prof basically said, if given one of the following three, you can find the other two:

  • Cumulative distribution function
  • Moment Generating Function
  • Probability Density Function

But my econometrics professor said CDFs are more fundamental than PDFs because there are examples where you can have a CDF but the PDF isn't defined.

Are CDFs more fundamental than PDFs? How do I know whether a PDF or a MGF can be derived from a CDF?

$\endgroup$
  • 23
    $\begingroup$ Is this some kind of fundamentality contest? Do we have a panel of celebrity judges? All these three concepts can be used to define a measure on a space $\mathbb{R}^d$. However for a given CDF, MGF and PDF may not exist, as PDF is defined as a derivative of CDF, and MGF is defined as a $\int_{\mathbb{R}}\exp(tx)dF(x)$, and this integral need not exist. However this does not mean that any of these concepts is less fundamental. Fundamental is a nice adjective which has no mathematical definition. It is a synonym of important. $\endgroup$ – mpiktas May 25 '16 at 6:56
  • 6
    $\begingroup$ @mpiktas: Every probability distribution on (a subset of) $\mathbb R^n$ has a CDF, and it uniquely defines the distribution. Not all probability distributions have a PDF or an MGF, however (but they do all have a characteristic function). $\endgroup$ – Ilmari Karonen May 25 '16 at 7:51
  • 3
    $\begingroup$ @mpiktas You might do it with $\mathcal A=\{\mathbb R,\varnothing\}$ on $\mathbb R$. Then $P((-\infty,x])$ is not defined. Nevertheless it is crystal clear to me why the professor used the expression "more fundamental". The adjective might have no well defined mathematical meaning, but so what? I speak (some) English too. Every PDF that we know of has an underlying CDF. Here "underlying" has a nice association with "fundamental". The opposite is not true. $\endgroup$ – drhab May 25 '16 at 10:42
  • 2
    $\begingroup$ @drhab, naturally I was talking about the Radon-Nikodym derivative :) I too perfectly understand what professor had in mind, but in my opinion it is dangerous to use such expressions with students, because then instead of trying to understand the difference between the mathematical concepts they try to rank them according to fundamentality, which is fundamentaly wrong. Pun intended. $\endgroup$ – mpiktas May 25 '16 at 12:40
  • 4
    $\begingroup$ @mpiktas: sure, there‘s no precise definition of “fundamental”. But there’s a big middle ground between “rigorously defined” and “totally meaningless”. In our mathematics itself, of course, everything must in the end be completely rigorous, so we get very accustomed to slapping down anything that isn’t. But when we talk and think about mathematics, we have subjective-yet-meaningful notions like “fundamental”, “general”, etc as well, just like everyone else; and that’s OK. $\endgroup$ – PLL May 27 '16 at 12:34
68
+100
$\begingroup$

Every probability distribution on (a subset of) $\mathbb R^n$ has a cumulative distribution function, and it uniquely defines the distribution. So, in this sense, the CDF is indeed as fundamental as the distribution itself.

A probability density function, however, exists only for (absolutely) continuous probability distributions. The simplest example of a distribution lacking a PDF is any discrete probability distribution, such as the distribution of a random variable that only takes integer values.

Of course, such discrete probability distributions can be characterized by a probability mass function instead, but there are also distributions that have neither and PDF or a PMF, such as any mixture of a continuous and a discrete distribution:

Diagram of continuous, discrete and mixed probability distributions
(Diagram shamelessly stolen from Glen_b's answer to a related question.)

There are even singular probability distributions, such as the Cantor distribution, which cannot be described even by a combination of a PDF and a PMF. Such distributions still have a well defined CDF, though. For example, here is the CDF of the Cantor distribution, also sometimes called the "Devil's staircase":

Cantor distribution CDF
(Image from Wikimedia Commons by users Theon and Amirki, used under the CC-By-SA 3.0 license.)

The CDF, known as the Cantor function, is continuous but not absolutely continuous. In fact, it is constant everywhere except on a Cantor set of zero Lebesgue measure, but which still contains infinitely many points. Thus, the entire probability mass of the Cantor distribution is concentrated on this vanishingly small subset of the real number line, but every point in the set still individually has zero probability.


There are also probability distributions that do not have a moment-generating function. Probably the best known example is the Cauchy distribution, a fat-tailed distribution which has no well-defined moments of order 1 or higher (thus, in particular, having no well-defined mean or variance!).

All probability distributions on $\mathbb R^n$ do, however, have a (possibly complex-valued) characteristic function), whose definition differs from that of the MGF only by a multiplication with the imaginary unit. Thus, the characteristic function may be regarded as being as fundamental as the CDF.

$\endgroup$
  • $\begingroup$ You say that every distribution has CDF, but not every has PDF, but actually there are distributions that have PDFs and do not have closed-form CDFs e.g. multivariate normal. $\endgroup$ – Tim May 25 '16 at 20:45
  • 13
    $\begingroup$ @Tim: That's true, but only with the "closed form" qualifier; the CDF still exists, even if we can't write it in closed form. And in any case, the definition of a "closed form expression" is notoriously fuzzy; by some strict definitions, even the univariate normal distribution doesn't have a closed-form CDF, but if you consider the error function to be closed-form, it does. $\endgroup$ – Ilmari Karonen May 25 '16 at 21:13
  • 11
    $\begingroup$ @Tim It's not a counter-example. It's an arbitrary property that you chose as being important/fundamental for you. For me, the "exists" property is more important than "has closed form". More so, "always exists" versus "might sometimes not have closed form, just like any function". $\endgroup$ – Ark-kun May 26 '16 at 0:42
  • 3
    $\begingroup$ More important than the Cantor set has "infinitely many points" is that it has uncountably many points. It has the same cardinality is the interval $[0,1] \subset \Bbb{R}$. So it is an uncountable subset of the interval with zero (Lebesgue) measure. The measure you're describing is similarly interesting in that it is zero on countable subsets of the Cantor set and only nonzero on (some) uncountable subsets... $\endgroup$ – Eric Towers May 27 '16 at 7:42
  • 1
    $\begingroup$ @Ark-kun I'm playing devils advocate in here since there are cases where PDF is something more "directly available" then CDF. I like this answer (+1), but IMHO, this is something that also could be mentioned. $\endgroup$ – Tim May 27 '16 at 7:50
16
$\begingroup$

I believe your econometrics professor was thinking something along the following lines.

Consider the function $F$ with domiain $[0, 1]$ defined by

$$F(x) = \frac{1}{2}x \ \text{for} \ x < \frac{1}{2} $$ $$F(x) = \frac{1}{2}x + \frac{1}{2} \ \text{for} \ x \geq \frac{1}{2} $$

This is a discontinuous function, but a completely valid CDF for some probability distribution on $[0, 1]$. Note that, using this distribution

$$ P\left(\left\{\frac{1}{2}\right\}\right) = \frac{1}{2} $$

There is no function $f$ that serves as a PDF for this distribution, even though there is a CDF.

It's easy enough to check this is true in this simple example if you've seen this kind of thing before. Suppose there is such a pdf $f$, we will show it must have an impossible property, and hence cannot exist.

By the definition of a PDF, we must have

$$ \int_0^x f(t) dt = F(x) - F(0) = \frac{1}{4}x $$

for all $0 < x < \frac{1}{2}$. A function that integrates to a linear function must be constant (technically constant almost everywhere), so we conclude that

$$ f(x) = \frac{1}{4} \ \text{for} \ x < \frac{1}{2} $$

In the same way, but integrating starting at one, moving towards zero, and ending at $x > \frac{1}{2}$, we reach the same conclusion

$$ f(x) = \frac{1}{4} \ \text{for} \ x > \frac{1}{2} $$

So we have determined $f$ everywhere except for $f\left(\frac{1}{2}\right)$. But it really does not matter what $f\left(\frac{1}{2}\right)$ is, it cannot have the integration property desired. Since

$$ P\left(\left\{\frac{1}{2}\right\}\right) = \frac{1}{2} $$

we would need

$$ \int_{\frac{1}{2} - \epsilon}^{\frac{1}{2} + \epsilon} f(t) dt > \frac{1}{2} $$

for every interval containing $\frac{1}{2}$. But in fact the value of any integral is unaffected by changing the value of a function at any single point, so

$$ \int_{\frac{1}{2} - \epsilon}^{\frac{1}{2} + \epsilon} f(t) dt = \int_{\frac{1}{2} - \epsilon}^{\frac{1}{2} + \epsilon} \frac{1}{4} dt = \frac{1}{2} \epsilon $$

So there's no way out, a function such as $f$ cannot exist.

You can recover the spirit of a PDF, but you must use more sophisticated mathematical objects, either a measure or a distribution.

$\endgroup$
  • 3
    $\begingroup$ This "impossible property" is easily achieved by summing $\frac{1}{2} \delta \left(x-\frac{1}{2}\right)$ into an otherwise quite conventional PDF, where $\delta(x)$ is the Dirac delta, a generalized function with value 0 everywhere except an (infinitely-tall) "spike" at $x=0$, with the special property that $$\int\limits_{-\infty}^{+\infty} \delta(x) \,\mathrm{d}x = 1$$. $\endgroup$ – Iwillnotexist Idonotexist May 25 '16 at 11:11
  • 2
    $\begingroup$ @iwill A PDF, by definition, is an equivalence class of functions (in the $L^1$ norm with respect to Lebesgue measure). The Dirac delta does not qualify--which is precisely why it has to be called a "generalized function." $\endgroup$ – whuber May 25 '16 at 13:52
  • $\begingroup$ @IwillnotexistIdonotexist What whuber said is what i was hinting at in the last line. I used the word "distribution". $\endgroup$ – Matthew Drury May 25 '16 at 14:03
  • 4
    $\begingroup$ Your example does not have a pdf only because you implicitly assume the dominating measure is the Lebesgue measure. But it does when you use a dominating measure that includes a point mass at $1/2$, for instance the sum of the Lebesgue measure and of the Dirac at $1/2$. $\endgroup$ – Xi'an May 26 '16 at 11:46
4
$\begingroup$

Ilmari gives a good answer from a theoretical perspective. However, one may also ask what purposes the density (pdf) and the distribution function (pdf) serve for practical computations. This could clarify for which situations one is more directly useful than the other.

For probability distributions on $\mathbb{R}$ the distribution function directly gives probabilities of all intervals $(-\infty,x]$. From these probabilities, the probability of a finite union of intervals can be computed by elementary arithmetic. For all practical purposes I can think of, these are the only probabilities you would like to be able to compute. It may be theoretically convenient to express these $-$ or probabilities of more general sets $-$ as integrals, but for the actual computation we effectively need the distribution function.

The density is, however, essential for statistics, as the likelihood is defined in terms of the density. Thus if we want to compute the maximum likelihood estimate, we directly need the density.

If we turn to the comparison of an empirical and a theoretical distribution, both can be useful, but methods such as pp- and qq-plots based on the distribution function are often preferred.

For probability distributions on $\mathbb{R}^d$ for $d \geq 2$ the distribution function plays a less prominent role. One reason is that the probabilities for many sets of interest (balls, ellipsoids, cones etc.) cannot easily be computed from it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.