3
$\begingroup$

If we have a probability density function, we can integrate it and now we have a cumulative probability distribution function. Can you help my intuition on why this works?

$\endgroup$
5
  • 1
    $\begingroup$ You're already familiar with interpreting the CDF as the area under the distribution's PDF ("bell curve" in the normal case), yes? $\endgroup$ – J. M. is not a statistician Mar 22 '17 at 15:01
  • $\begingroup$ Yes. An answer expanding from that idea with visual examples would be nice. $\endgroup$ – Atte Juvonen Mar 22 '17 at 15:33
  • $\begingroup$ are you familiar with measure theory at all? The clearest way to understand why integrating a density gives a probability is through that lens, but it won't be very meaningful if it's all new to you $\endgroup$ – jld Mar 22 '17 at 15:37
  • $\begingroup$ Not familiar with measure theory. $\endgroup$ – Atte Juvonen Mar 22 '17 at 16:03
  • $\begingroup$ en.wikipedia.org/wiki/Fundamental_theorem_of_calculus $\endgroup$ – Mark L. Stone Mar 25 '17 at 19:11
7
$\begingroup$

Loosely speaking...

Integration can be loosely thought as an analog to summation. Imagine that you had a discrete random variable following the categorical distribution, with $\Pr(X = x_i) = p_i$ for $i = 1,\dots,n$ and $\sum_i p_i = 1$. In such case $\Pr(X\le1) = p_1$, what is pretty obvious, and $\Pr(X\le2) = p_1 +p_2$, in more general terms

$$ \Pr(X \le k) = \sum_{j=1}^k p_j $$

so

$$ F(k) = \sum_{j=1}^k f(j) $$

Now recall that in continuous random variables we have infinitely many $x$'s, so $\Pr(X=x) = 0$ and because of that we use probability density functions, that tell us about "probabilities per foot". Imagine that $X$ is a continuous random variable. Imagine that you bin the values of $X$ in the $[x_i, x_i+\Delta x]$ bins, now given the axioms of probability it follows that probabilities of all the intervals need to sum to unity

$$ \sum_i \Pr([x_i, x_i+\Delta x]) = 1 $$

this can be re-refined in terms of probability densities (probabilities per unit), as described by Kruschke (2015), as

$$ \sum_i \frac{\Pr([x_i, x_i+\Delta x])}{\Delta x} \Delta x = 1 $$

now as $\Delta x \to 0$ this becomes an integral

$$ \int f(x) \, d x = 1 $$

So in continuous case we use integration to "sum" the probability densities up to a given value

$$ \Pr(X \le x) = F(x) = \int_{-\infty}^x f(t) \, dt $$

To learn more you could check the Wikipedia page on integrals, Khan academy videos on Riemann approximation of integrals and chapter 4 of the book by Kruschke

Kruschke, J.K. (2015). Doing Bayesian Data Analysis: A Tutorial with R, JAGS, and Stan. Elsevier.

since all of them pay pretty much attention to providing an intuitive explanation of integration.

$\endgroup$
1
5
$\begingroup$

Maybe a visual example would help with intuition, as an adjunct to the mathematical discussion in @Tim's answer. The example below uses R code. We create two empirical probability density distributions by simulating random draws from normal distributions, then plot both the probability density functions (in black) and cumulative probability density functions (in red).

The red lines show, at any given point along the x-axis, the cumulative probability density (that is, the integral of the probability density) from negative infinity to the current x-value. This is the numerical integration equivalent of $\Pr(X \le x) = \int_{-\infty}^x f(t) \, dt$ from @Tim's answer.

The red curve rises as we go from left to right because we continue to add (i.e., integrate) more of the density. The slope of the red curve is steepest where the black curve is highest, because that's where each increment along the x-axis adds the most additional area under the curve.

In the right-hand graph, note that in between the first and second "humps", the red curve's y-value is about 0.33. That's because we have 1,500 total draws from the two normal distributions, and 500 of them (i.e., one-third of the total) are from the normal with mean=10 and standard deviation=0.6. Thus, after integrating the area under the first hump, we've added up one-third of the total probability density under the black curve (i.e., one-third of the area under the curve).

When we've reached the right edge of the plot, we've added up (i.e., integrated) all of the area under the black probability density curve, which is why the red curve rises to a value of 1.0 at the right edge of each plot.

par(mfrow=c(1,2))

# Standard normal distribution
set.seed(5)
dens = density(rnorm(1000,0,1))

plot(dens$x, dens$y, type="l", ylim=c(0,1), main="Normal Distribution", ylab="Density")
lines(dens$x, cumsum(dens$y)*diff(dens$x)[1], col="red")

# Combination of two normal densities with mean 10 and 15 and standard deviation 
#  0.6 and 0.3, respectively; 500 draws from the first and 1,000 draws from the second
set.seed(5)
dens = density(c(rnorm(500,10,0.6), rnorm(1000,15,0.3)))

plot(dens$x, dens$y, type="l", ylim=c(0,1), main="Two normals combined", ylab="")
lines(dens$x, cumsum(dens$y)*diff(dens$x)[1], col="red")

enter image description here

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.