27
$\begingroup$

Conceptually I grasp the meaning of the phrase "the total area underneath a PDF is 1". It should mean that the chances of the outcome being in the total interval of possibilities is 100%.

But I cannot really understand it from a "geometric" point of view. If, for instance, in a PDF the x-axis represents length, would the total area underneath the curve not become larger if x was measured in mm rather than km?

I always try to picture how the area underneath the curve would look if the function were flattened to a straight line. Would the height (position on the y-axis) of that line be the same for any PDF,or would it have a value contingent on the interval on the x-axis for which the function is defined?

$\endgroup$
10
  • $\begingroup$ You could change the scale of $x$ axis from km to mm but what would that amount to? You would still have the exact same picture and six more zeros for units at the $x$ axis. You could zoom in or zoom out if you wish, but that would not change the picture. Meanwhile, if the pdf curve is a straight horizontal line (which implies uniform distribution), its position on the $y$ axis does not depend on the units of the $x$ axis but only on the length of the interval on the $x$ axis. Not sure how helpful it is for you, but for me the idea of zooming in and out makes it easier to understand. $\endgroup$ Jan 14, 2015 at 12:58
  • 2
    $\begingroup$ That seems to be true. But that's kind of using a (admittedly weird) magnifying glass that magnifies in horizontal direction by 1000 and at the same time shrinks proportionally in the vertical direction. But the essence of the picture will not change if you only change the scale. $\endgroup$ Jan 14, 2015 at 13:22
  • 2
    $\begingroup$ This question seems to me to be the same as the one asked (in a different way) and answered at stats.stackexchange.com/questions/4220/…. $\endgroup$
    – whuber
    Jan 14, 2015 at 15:14
  • 1
    $\begingroup$ @amoeba, Yes, while many may feel compelled to vote for the longer answer in recognition of the effort put into it (which I did as well, btw), Aksakal answered my question much more clearly and succinctly. To be fair I would say Silverfish's answer also helped and would come in at a close second. $\endgroup$
    – TheChymera
    Jan 16, 2015 at 14:53
  • 2
    $\begingroup$ @amoeba A completely different direction of answer could have been to focus on the fact that PDFs are derivatives of CDFs, so the area under the PDF is simply the limiting value of the CDF - which is clearly one, regardless of the units used. I was tempted to include a short section on this but felt my answer was long enough already (and besides, the key to the OP's issue seemed to be the units issue, which the CDF approach rather skirts around). $\endgroup$
    – Silverfish
    Jan 18, 2015 at 0:09

4 Answers 4

16
$\begingroup$

Probability density function is measured in percentages per unit of measure of your x-axis. Let's say at a given point $x_0$ your PDF is equal to 1000. This means that the probability of $x_0<x<x_0+dx$ is $1000\,dx$ where $dx$ is in meters. If you change the units to centimeters, then the probability should not change for the same interval, but the same interval has 100 more centimeters than meters, so $1000\,dx=PDF'(x_0')\cdot100\,dx'$ and solving we get $PDF'(x_0')=\frac{PDF(x_0)}{100}$. There's 100 times less units of probability (percentages) per centimeter than per meter.

$\endgroup$
53
+150
$\begingroup$

It might help you to realise that the vertical axis is measured as a probability density. So if the horizontal axis is measured in km, then the vertical axis is measured as a probability density "per km". Suppose we draw a rectangular element on such a grid, which is 5 "km" wide and 0.1 "per km" high (which you might prefer to write as "km$^{-1}$"). The area of this rectangle is 5 km x 0.1 km$^{-1}$ = 0.5. The units cancel out and we are left with just a probability of one half.

If you changed the horizontal units to "metres", you'd have to change the vertical units to "per metre". The rectangle would now be 5000 metres wide, and would have a density (height) of 0.0001 per metre. You're still left with a probability of one half. You might get perturbed by how weird these two graphs will look on the page compared to each other (doesn't one have to be much wider and shorter than the other?), but when you're physically drawing the plots you can use whatever scale you like. Look below to see how little weirdness need be involved.

You might find it helpful to consider histograms before you move on to probability density curves. In many ways they are analogous. A histogram's vertical axis is frequency density [per $x$ unit] and areas represent frequencies, again because horizontal and vertical units cancel out upon multiplication. The PDF curve is a sort of continuous version of a histogram, with total frequency equal to one.

An even closer analogy is a relative frequency histogram - we say such a histogram has been "normalized", so that area elements now represent proportions of your original data set rather than raw frequencies, and the total area of all the bars is one. The heights are now relative frequency densities [per $x$ unit]. If a relative frequency histogram has a bar that runs along $x$ values from 20 km to 25 km (so the width of the bar is 5 km) and has a relative frequency density of 0.1 per km, then that bar contains a 0.5 proportion of the data. This corresponds exactly to the idea that a randomly chosen item from your data set has a 50% probability of lying in that bar. The previous argument about the effect of changes in units still applies: compare the proportions of data lying in the 20 km to 25 km bar to that in the 20,000 metres to 25,000 metres bar for these two plots. You might also confirm arithmetically that the areas of all bars sum to one in both cases.

Relative frequency histograms with different units

What might I have meant by my claim that the PDF is a "sort of continuous version of a histogram"? Let's take a small strip under a probability density curve, along $x$ values in the interval $[x, x + \delta x]$, so the strip is $\delta x$ wide, and the height of the curve is an approximately constant $f(x)$. We can draw a bar of that height, whose area $f(x) \, \delta x$ represents the approximate probability of lying in that strip.

How might we find the area under the curve between $x=a$ and $x=b$? We could subdivide that interval into little strips and take the sum of the areas of the bars, $\sum f(x) \, \delta x$, which would correspond to the approximate probability of lying in the interval $[a,b]$. We see that the curve and the bars do not precisely align, so there is an error in our approximation. By making $\delta x$ smaller and smaller for each bar, we fill the interval with more and narrower bars, whose $\sum f(x) \, \delta x$ provides a better estimate of the area.

To calculate the area precisely, rather than assuming $f(x)$ was constant across each strip, we evaluate the integral $\int_a^b f(x) dx$, and this corresponds to the true probability of lying in the interval $[a,b]$. Integrating over the whole curve gives a total area (i.e. total probability) one, for the same reason that summing up the areas of all the bars of a relative frequency histogram gives a total area (i.e. total proportion) of one. Integration is itself a sort of continuous version of taking a sum.

enter image description here

R code for plots

require(ggplot2)
require(scales)
require(gridExtra)
# Code for the PDF plots with bars underneath could be easily readapted

# Relative frequency histograms
x.df <- data.frame(km=c(rep(12.5, 1), rep(17.5, 2), rep(22.5, 5), rep(27.5, 2)))
x.df$metres <- x.df$km * 1000

km.plot <- ggplot(x.df, aes(x=km, y=..density..)) +
  stat_bin(origin=10, binwidth=5, fill="steelblue", colour="black") +
  xlab("Distance in km") + ylab("Relative frequency density per km") +
  scale_y_continuous(minor_breaks = seq(0, 0.1, by=0.005))

metres.plot <- ggplot(x.df, aes(x=metres, y=..density..)) +
  stat_bin(origin=10000, binwidth=5000, fill="steelblue", colour="black") +
  xlab("Distance in metres") + ylab("Relative frequency density per metre") +
  scale_x_continuous(labels = comma) +
  scale_y_continuous(minor_breaks = seq(0, 0.0001, by=0.000005), labels=comma)

grid.arrange(km.plot, metres.plot, ncol=2)
x11()

# Probability density functions
x.df <- data.frame(x=seq(0, 1, by=0.001))
cutoffs <- seq(0.2, 0.5, by=0.1) # for bars
barHeights <- c(0, dbeta(cutoffs[1:(length(cutoffs)-1)], 2, 2), 0) # uses left of bar

x.df$pdf <- dbeta(x.df$x, 2, 2)
x.df$bar <-  findInterval(x.df$x, cutoffs) + 1 # start at 1, first plotted bar is 2
x.df$barHeight <- barHeights[x.df$bar]

x.df$lastBar <- ifelse(x.df$bar == max(x.df$bar)-1, 1, 0) # last plotted bar only
x.df$lastBarHeight <- ifelse(x.df$lastBar == 1, x.df$barHeight, 0)
x.df$integral <- ifelse(x.df$bar %in% 2:(max(x.df$bar)-1), 1, 0) # all plotted bars
x.df$integralHeight <- ifelse(x.df$integral == 1, x.df$pdf, 0)

cutoffsNarrow <- seq(0.2, 0.5, by=0.025) # for the narrow bars
barHeightsNarrow <- c(0, dbeta(cutoffsNarrow[1:(length(cutoffsNarrow)-1)], 2, 2), 0) # uses left of bar
x.df$barNarrow <-  findInterval(x.df$x, cutoffsNarrow) + 1 # start at 1, first plotted bar is 2
x.df$barHeightNarrow <- barHeightsNarrow[x.df$barNarrow]

pdf.plot <- ggplot(x.df, aes(x=x, y=pdf)) +
  geom_area(fill="lightsteelblue", colour="black", size=.8) +
  ylab("probability density") +
  theme(panel.grid = element_blank(),
  axis.text.x = element_text(colour="black", size=16))

pdf.lastBar.plot <- pdf.plot +
  scale_x_continuous(breaks=tail(cutoffs, 2), labels=expression(x, x+delta*x)) +
  geom_area(aes(x=x, y=lastBarHeight, group=lastBar), fill="steelblue", colour="black", size=.8) +
  annotate("text", x=0.73, y=0.22, size=6, label=paste("P(paste(x<=X)<=x+delta*x)%~~%f(x)*delta*x"), parse=TRUE)

pdf.bars.plot <- pdf.plot +
  scale_x_continuous(breaks=cutoffs[c(1, length(cutoffs))], labels=c("a", "b")) +
  geom_area(aes(x=x, y=barHeight, group=bar), fill="steelblue", colour="black", size=.8) +
  annotate("text", x=0.73, y=0.22, size=6, label=paste("P(paste(a<=X)<=b)%~~%sum(f(x)*delta*x)"), parse=TRUE)

pdf.barsNarrow.plot <- pdf.plot +
  scale_x_continuous(breaks=cutoffsNarrow[c(1, length(cutoffsNarrow))], labels=c("a", "b")) +
  geom_area(aes(x=x, y=barHeightNarrow, group=barNarrow), fill="steelblue", colour="black", size=.8) +
  annotate("text", x=0.73, y=0.22, size=6, label=paste("P(paste(a<=X)<=b)%~~%sum(f(x)*delta*x)"), parse=TRUE)

pdf.integral.plot <- pdf.plot +
  scale_x_continuous(breaks=cutoffs[c(1, length(cutoffs))], labels=c("a", "b")) +
  geom_area(aes(x=x, y=integralHeight, group=integral), fill="steelblue", colour="black", size=.8) +
  annotate("text", x=0.73, y=0.22, size=6, label=paste("P(paste(a<=X)<=b)==integral(f(x)*dx,a,b)"), parse=TRUE)

grid.arrange(pdf.lastBar.plot, pdf.bars.plot, pdf.barsNarrow.plot, pdf.integral.plot, ncol=2)
$\endgroup$
7
  • 5
    $\begingroup$ @PatrickT Thanks - teaching experience suggests that sometimes you've got to try a couple of things before the penny drops, because different learners (or readers) come in with different levels of knowledge. The first two lines should convince any reader who knows dimensional analysis (for instance if the studied physical sciences or engineering) but I'm hoping the plots sort out the rest! In my experience the histogram approach works well for students who have come across them before; the gap between "relative frequency density" and "probability density" is easier to bridge than $f(x)=F'(x)$. $\endgroup$
    – Silverfish
    Jan 14, 2015 at 17:45
  • 1
    $\begingroup$ @Silverfish: This is the first time I've seen someone use the term "penny drops" in English! $\endgroup$
    – user541686
    Jan 14, 2015 at 22:01
  • 3
    $\begingroup$ The first chart looks like someone flipping the bird :) $\endgroup$
    – Aksakal
    Jan 15, 2015 at 16:09
  • 1
    $\begingroup$ @Aksakal Ooof. I didn't notice that. Must remember not to use that example in class without a few modications. (On the same lines, when making up a problem to solve on the board, there are certain numbers like 69 I try to avoid appearing. Experience is hard-won.) $\endgroup$
    – Silverfish
    Jan 15, 2015 at 16:14
  • 1
    $\begingroup$ @Aksakal: Wow, thanks for using the expression "flipping the bird"! I learned something new today (after googling it). $\endgroup$
    – amoeba
    Jan 15, 2015 at 19:35
8
$\begingroup$

You already got two answers, with an excellent one by Silverfish, however I feel that an illustration could be useful in here since you asked about geometry and "imagining" yourself those functions.

Lets start with a simple example of Bernoulli distribution:

$$ f(x) = \begin{cases} p & \text{if }x=1, \\[6pt] 1-p & \text {if }x=0.\end{cases} $$

enter image description here

Since the values are discrete there is no "curve" but only two points, however the idea is similar: if you want to know total probability (area under the curve) you have to sum up probabilities of both possible outcomes:

$$p + (1 - p) = 1$$

There is only $p$ and $1-p$ in this equation since we have only two possible point outcomes with a given probabilities.

The same would be with Poisson distribution that is also a discrete probability distribution. There are more than two values, so you can imagine that there is a line that connects the points, however to compute the total probability you would have to sum up all the probabilities of $x$'s. Poisson distribution is often used to describe count data, so you can think of it as each $x$ is a number of certain events and $f(x)$ is a probability of this outcome. You could imagine that each point on the plot below is actually a height of a stack made of some outcomes: $x_1$ is a stack of all the "$x_1$" outcomes that you observed etc. The total "area under the curve" would be here all the stacks summed up (or a meta-stack of all the outcomes) but since we do not sum up numbers of occurrences but rather probabilities, they sum up to $1$. So you should not think of it as sum of counts $\sum \#\{x_i\}=N$, but rather as sum of probabilities: $\sum \#\{x_i\}/N=1$ where $N$ is a total number of all possible outcomes.

enter image description here

Now let's consider a normal distribution that actually is a continuous distribution - so we don't have "points" since the values of $x$ are on continuous scale, i.e. there is infinitely many values of $x$. So if there were points you couldn't see them no matter how much would you "zoom in", since there always could be some an infinite number of smaller points between any given points. Because of that here we actually have a curve - you can imagine that it is made of infinitely many "points". You could ask yourself: how to compute a sum of infinite number of probabilities..? On the plot below red curve is a normal PDF and the black boxes is histogram of some values drawn from the distribution. So histogram plot has simplified our distribution to the finite number of "boxes" with a certain width and if you summed up the heights of the boxes multiplied by their width you would end up with an area under the curve - or area of all the boxes. We use areas rather points in here since each box is a summary of an infinite number of "points" that were packed up in the box.

enter image description here

So to get total area we take the heights (i.e. $f(x)$) and widths (e.g. first box has width: $-2.5 - -3 = 0.5$, the same as all the other boxes). In the actual figure plotted the heights of the boxes are:

0.010 0.028 0.094 0.198 0.260 0.400 0.404 0.292 0.166 0.092 0.044 0.010 0.002

if you sum them up multiplying each by $0.5$ (width), they will sum up to $1$. In here you cannot count anything since there is infinitely many possible points that form the curve. On another hand, since we are talking about probabilities, the probability of all the possible outcomes has to be $1$.

In this case we use "probability per unit" and the unit can have any width of your choice. Consider "all possible outcomes" on the continuous scale as a line that could be divided into the parts, and each part could be divided into some smaller parts up to infinitely small ones. The total probability of this line is $1$. If it would be flat than you could imagine that it's total length is $1$ and by dividing it you get probabilities of the parts. If the line is not flat, the probability per part is described by function $f(x)$. So the units actually doesn't matter since there is infinite number of possible "points" it is probability per unit, where unit is always the same: a fraction of "total" length.

This approach illustrates in a simplified way a little bit more complicated issue - taking integrals. In continuous case you use integrals for calculating the area under the curve. Integral of the area of the curve between points $a$ and $b$ ($-3$ and $3$ on out plot) is:

$$\int_a^b \! f(x)\,dx$$

where $f(x)$ is height and $dx$ is width and you could think of $\int$ as $\sum$ for continuous variables. For learning more on integrals and calculus you could check the Khan Academy lectures.

You asked also about the "flat" (uniform) distribution:

enter image description here

First notice that this is not a valid uniform distribution since it should have parameters such that $-\infty < a < b < \infty$, so to integrate to $1$. If you think of it, it is continuous and since it is flat, it is some kind of box with a width from $-\infty$ to $\infty$. If you wanted to calculate area of such box, you would be multiplying the height by width. Unfortunately, while the width is infinitely wide, for it to integrate to $1$ the height would have to be some $\varepsilon$ that is enormously small... So this is a complicated case and you could imagine it rather in abstract terms. Notice that, as Ilmari Karonen noticed in the comment, this is rather an abstract idea that is not really possible in practice (see the comment below). If using such distribution as a prior, it would be an improper prior.

Notice that in the continuous case probability density function gives you density estimates rather then probabilities, so heights (or their sum) could exceed $1$ (see here for more).

$\endgroup$
2
  • 1
    $\begingroup$ For the flat (i.e. uniform) distribution on an infinite line, "complicated" really means "impossible": a uniform distribution on set of infinite measure is not a valid probability distribution, precisely because it can't be scaled to integrate to $1$. It's sometimes useful to pretend that it is one, but taking this too far quickly leads to paradoxes. (E.g. let $X$ and $Y$ be independent and uniformly distributed on $(-\infty,\infty)$; what is the probability that $|X|<|Y|$?) $\endgroup$ Jan 15, 2015 at 1:04
  • $\begingroup$ You are right, "complicated" is too informal. I'll make corrections later on. $\endgroup$
    – Tim
    Jan 15, 2015 at 5:47
0
$\begingroup$

The following key idea was mentioned in a comment, but not in an existing answer...

One way of intuiting about the properties of a PDF is to consider that the PDF and the CDF are related by integration (calculus) -- and that the CDF has a monotonic output representing a probability value between 0 and 1.

The unitless integrated total area under the PDF curve is not affected by X-axis units.

To put it simply...

Area = Width x Height

If the X-axis gets larger, numerically, due to a change in units, then the Y-axis must become smaller by a corresponding linear factor.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.