7
$\begingroup$

Can you please explain those three terms and the relationship between them (both graphical and mathematical way would be fine)?

EDIT: Those terms are mainly associated with functions.The quotation from this article says:

The probability density function, pdf as f(x). (Note: This function is also known as the probability distribution function and the probability mass function, but will be referred to henceforth as the probability density function.)

According to above:

Probability density function = Probability distribution function = Probability mass function

But wikipedia has separate articles about Probability Mass Function and Probability Density Function. And here is mentioned that the name Probability Distribution Function depends on the context in which is used. So where is the truth?

For completeness the term "cumulative density" is wrong as it was discussed bellow.

$\endgroup$
8
  • $\begingroup$ I have never heard the expression "cumulative density". Can you show an example where it is used? $\endgroup$
    – user765195
    Jun 19, 2015 at 0:52
  • $\begingroup$ E.g. here ncbi.nlm.nih.gov/pubmed/23543120 $\endgroup$ Jun 19, 2015 at 0:55
  • $\begingroup$ Now I've found on wikipedia en.wikipedia.org/wiki/Cumulative_density_function on this: Cumulative density function is a self-contradictory phrase resulting from confusion between: probability density function and cumulative distribution function. The two words cumulative and density contradict each other. $\endgroup$ Jun 19, 2015 at 0:57
  • $\begingroup$ Yes, cumulative density function is meaningless. I believe the article you linked to is trying to talk about cumulative distribution function and by mistake calls it cumulative density function. $\endgroup$
    – user765195
    Jun 19, 2015 at 0:59
  • $\begingroup$ To avoid confusion, I removed the incorrect term from your title. $\endgroup$
    – user765195
    Jun 19, 2015 at 1:01

2 Answers 2

6
$\begingroup$

Looking at the links you provided, and what I'v seen in the past, there seems to be a lot of different names for what is essentially a particular function and its derivative.

First, let's cross off one of the names you mention in the title: cumulative density function. That just doesn't make sense. A density function concerns itself with local properties of a phenomenon, while a cumulative function would be looking at global properties. A parallel term in calculus perhaps would be "integral derivative function", which makes no sense. So, the term "cumulative density" is just incorrect.

Now let's deal with the remaining terms: cumulative distribution function, distribution function, probability density function, and probability mass function.

The terms cumulative distribution function, probability density function, and probability mass function have unique meanings, which I will try to explain below.

I can't remember seeing the term "distribution function" being used as an equivalent to "probability density function" and "probability mass function", but it doesn't mean it is not used, considering that so many different disciplines use these concepts. But in measure theoretical probability, the term "distribution function" always refers to "cumulative distribution function", and the "cumulative" part is always dropped.

Next, let's define the terms and see what is their relationship. If you really want a truly complete answer, you'll need to know some measure theory, but I'm going to try to give a reasonable answer without using any measure theory. Nevertheless, a good understanding of calculus is inevitable.

If $X$ is a random variable, then its cumulative distribution function (CDF) is a function $F_X$ defined on real numbers as follows: \begin{equation} F_X(x) = P(X \leq x) \end{equation}

It is not hard to see that $F_X$ is increasing: if $a < b$, then $F_x(a) \leq F_X(b)$. We can also show that $F_X$ is right-continuous at every $x$, meaning that if you approach $x$ through values larger than $x$, you'll get $F_X(x)$. In mathematical notation, this is written as \begin{equation} \lim_{z \to x^+} F_X(z) =F_X(x). \end{equation}

But $F_X$ can have jump points: points where it is discontinuous. Since $F_X$ is continuous from right everywhere, these discontinuities must be from the left. What that means is that if you approach such a point (say $a$) through values smaller than $a$, then the value of $F_X$ at those points does not approach the value of $F_X(a)$.

As a simple example, consider the random variable $X$ that always takes the value $3$. Then it is easy to see that $F_X(x) = 0$ for $x < 3$ and $F_X(x) = 1$ for $3 \leq x$. If you approach 3 via values larger than 3, then $F_X(x) = 1$ and the closer you get to 3, the value of $F_X(x)$ stays at 1. However, if you approach 3 through values less than $3$, then $F_X(x) = 0$ for these $x$, and regardless of how close to $3$ you get, you'll still get $F_X(x) = 0$. In mathematical notation, we write that as $F_X(3+) = F_X(3) = 1$ and $F_X(3-) = 0$.

In general, if we indicate by $F_X(a-)$ the value that $F_X(x)$ approaches to as $x$ approaches $a$ through values smaller than $a$, then the jump points of $F_X$ are those points $a$ such that $F_X(a) - F_X(a-) \neq 0.$ But remember that $F_X$ is increasing, and that implies that $F_X(a) - F_X(a-) > 0$ at jump points.

Another corollary of the increasing property of $F_X$ is that it can have at most countably many jump points. That means that we can put them in a list (though the list can be infinite). Let's assume that $a_1,a_2,\ldots,$ is the (possibly infinite) set of jump points of $F_X$. We will try to extract that part of $F_X$ that corresponds to the $a_i$s.

Define a point mass at $a$ to be the following function: \begin{equation} \delta_a(x) = \left\{ \begin{array}{ll} 0 & \mbox{if } x < a \\ 1 & \mbox{if } x \geq a. \end{array} \right. \end{equation}

Let $F_X(a_i) - F_X(a_i-) = b_i$. Define the function $F_X^d$ as follows. \begin{equation} F_X^d(x) = \sum_i b_i \delta_{a_i}(x). \end{equation}

If $F_X(x) = F^d_X(x)$ for all $x$, then $X$ is called a discrete random variable, and the function
\begin{equation} p_X(x) = \left\{ \begin{array}{ll} 0 & \mbox{if } x \neq a_1,a_2,\ldots \\ b_i & \mbox{if } x = a_i \qquad i = 1,2,\ldots \end{array} \right. \end{equation} is called the probability mass function of $X$.

If $F_X$ is differentiable function, with $F_X' = f$, then $f$ is called the probability density function. It is easy to see from the fundamental theorem of calculus that for any $a$ and $b$, \begin{equation} \int_a^b f(x)dx = F_X(b) - F_X(a) = P(a < X \leq b). \end{equation}

Notice that there is a vast sea between between when a probability mass function is defined ($F_X$ is a sum of point masses), and when a probability density function is defined ($F_X$ is differentiable). To understand what is in between, you need to study probability from a measure theoretical perspective.

$\endgroup$
4
  • $\begingroup$ I've edited the question, can you please check to see if your answer corresponds? Thank you $\endgroup$ Jun 19, 2015 at 23:09
  • $\begingroup$ I've updated the answer accordingly. $\endgroup$
    – user765195
    Jun 20, 2015 at 2:07
  • $\begingroup$ Thank you for your exhaustive reply. My math background is not as solid as yours. I understand that integration from x1 to x2 interval basically return the area between x axis and curve at given interval and derivation is basically inverse function and they are both used with continuous variables instead of summing. $\endgroup$ Jun 20, 2015 at 7:39
  • $\begingroup$ I mostly understand things from visual explanation. Now I know (I hope) difference between PDF vs CDF. Here are videos that helped me: shorter: goo.gl/fuegPo goo.gl/XewvXq goo.gl/s1IQlX longer (e.g. discusses various methods for converting PDF->CDF): goo.gl/rPibY4 I would like get my hands dirty and try to play with it (convert PDF to CDF,and vice versa, sum the area under PDF etc.) using python or R, but I do not know. Would you be kind enough to extend your answer to contain visual representation of what you've explained. Thank you very much. $\endgroup$ Jun 20, 2015 at 7:47
0
$\begingroup$

I see that the terminology is taking toll here. The basic concept behind those are very simple but takes one to visualize it properly.

First of all, PMF is individual probabilities of each discrete points. PDF is a tool or hack that we use in case of continuous random variable. Among other tasks (as highlighted in comment) CDF is also used for simulations where you need to generate data points from your choice of arbitrary complex PDF or PMF (but that's for later, here I'll say more about PDF). PDF is all you need for continuous r.v.

A rather more interesting question would be: why do we need PDF at all. Why is PMF not sufficient even for continuous r.v.? Actually at individual point, probability is zero (for continuous random variable), hence we take collective points and find group probability to make sense of what's going on in probabilistic world even if individual probability is zero. Density helps in that case. Other answers & comments here have rightly addressed this with mathematical derivations, but as mentioned it doesn't clarify deep seated doubt/curiosity. So without repeating the same verbose, here is youtube visualizations that build the concept from ground zero.

Why cannot we use Probability Mass Function (PMF) for continuous random variable?

What actually is Probability Density Function (PDF)?

$\endgroup$
4
  • 2
    $\begingroup$ Although many of these remarks are correct, some are wrong and misleading. The CDF is far more fundamental than a mere tool for simulation and the PDF is insufficient for doing much analysis of a continuous variable. Here on CV you can find many good expositions of the concepts that will not require readers to sit through a video. $\endgroup$
    – whuber
    Oct 2, 2020 at 15:45
  • $\begingroup$ @whuber Please provide link about CDF being more fundamental, that would be helpful, thanks. Secondly, I duly respect all good expositions here. However, the author requested for graphical explanations also, which I saw missing till now. I filled that gap, and should not be seen in negative light. Thanks. $\endgroup$
    – sroy
    Oct 2, 2020 at 15:57
  • 1
    $\begingroup$ I don't see any graphics in this post, nor can I even find a description of a graphic. As far as the CDF being more fundamental, you may refer to any theoretical text on statistics. The problems with the PDF include it doesn't necessarily exist and it has no good convergence properties. Examples: stats.stackexchange.com/a/314995/919; stats.stackexchange.com/a/86503/919. Perhaps the most prominent example is the Central Limit Theorem, which in its most elementary form (limit of Binomial distributions) cannot even be expressed in terms of PDFs. $\endgroup$
    – whuber
    Oct 2, 2020 at 16:15
  • $\begingroup$ @whuber It's animated graphics/video whose link I posted with descriptive names. These videos give intuitive rationale behind use of PDF. If you read authors comment in above post, he prefers such visualizations. He also mentioned about few links. BTW, thanks for providing link for CDF. $\endgroup$
    – sroy
    Oct 2, 2020 at 17:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.