"The total area underneath a probability density function is 1" - relative to what?

Question

Conceptually I grasp the meaning of the phrase "the total area underneath a PDF is 1". It should mean that the chances of the outcome being in the total interval of possibilities is 100%.

But I cannot really understand it from a "geometric" point of view. If, for instance, in a PDF the x-axis represents length, would the total area underneath the curve not become larger if x was measured in mm rather than km?

I always try to picture how the area underneath the curve would look if the function were flattened to a straight line. Would the height (position on the y-axis) of that line be the same for any PDF,or would it have a value contingent on the interval on the x-axis for which the function is defined?

Answer 1

Probability density function is measured in percentages per unit of measure of your x-axis. Let's say at a given point $x_0$ your PDF is equal to 1000. This means that the probability of $x_0<x<x_0+dx$ is $1000\,dx$ where $dx$ is in meters. If you change the units to centimeters, then the probability should not change for the same interval, but the same interval has 100 more centimeters than meters, so $1000\,dx=PDF'(x_0')\cdot100\,dx'$ and solving we get $PDF'(x_0')=\frac{PDF(x_0)}{100}$. There's 100 times less units of probability (percentages) per centimeter than per meter.

Answer 2

It might help you to realise that the vertical axis is measured as a probability density. So if the horizontal axis is measured in km, then the vertical axis is measured as a probability density "per km". Suppose we draw a rectangular element on such a grid, which is 5 "km" wide and 0.1 "per km" high (which you might prefer to write as "km$^{-1}$"). The area of this rectangle is 5 km x 0.1 km$^{-1}$ = 0.5. The units cancel out and we are left with just a probability of one half.

If you changed the horizontal units to "metres", you'd have to change the vertical units to "per metre". The rectangle would now be 5000 metres wide, and would have a density (height) of 0.0001 per metre. You're still left with a probability of one half. You might get perturbed by how weird these two graphs will look on the page compared to each other (doesn't one have to be much wider and shorter than the other?), but when you're physically drawing the plots you can use whatever scale you like. Look below to see how little weirdness need be involved.

You might find it helpful to consider histograms before you move on to probability density curves. In many ways they are analogous. A histogram's vertical axis is frequency density [per $x$ unit] and areas represent frequencies, again because horizontal and vertical units cancel out upon multiplication. The PDF curve is a sort of continuous version of a histogram, with total frequency equal to one.

An even closer analogy is a relative frequency histogram - we say such a histogram has been "normalized", so that area elements now represent proportions of your original data set rather than raw frequencies, and the total area of all the bars is one. The heights are now relative frequency densities [per $x$ unit]. If a relative frequency histogram has a bar that runs along $x$ values from 20 km to 25 km (so the width of the bar is 5 km) and has a relative frequency density of 0.1 per km, then that bar contains a 0.5 proportion of the data. This corresponds exactly to the idea that a randomly chosen item from your data set has a 50% probability of lying in that bar. The previous argument about the effect of changes in units still applies: compare the proportions of data lying in the 20 km to 25 km bar to that in the 20,000 metres to 25,000 metres bar for these two plots. You might also confirm arithmetically that the areas of all bars sum to one in both cases.

What might I have meant by my claim that the PDF is a "sort of continuous version of a histogram"? Let's take a small strip under a probability density curve, along $x$ values in the interval $[x, x + \delta x]$, so the strip is $\delta x$ wide, and the height of the curve is an approximately constant $f(x)$. We can draw a bar of that height, whose area $f(x) \, \delta x$ represents the approximate probability of lying in that strip.

How might we find the area under the curve between $x=a$ and $x=b$? We could subdivide that interval into little strips and take the sum of the areas of the bars, $\sum f(x) \, \delta x$, which would correspond to the approximate probability of lying in the interval $[a,b]$. We see that the curve and the bars do not precisely align, so there is an error in our approximation. By making $\delta x$ smaller and smaller for each bar, we fill the interval with more and narrower bars, whose $\sum f(x) \, \delta x$ provides a better estimate of the area.

To calculate the area precisely, rather than assuming $f(x)$ was constant across each strip, we evaluate the integral $\int_a^b f(x) dx$, and this corresponds to the true probability of lying in the interval $[a,b]$. Integrating over the whole curve gives a total area (i.e. total probability) one, for the same reason that summing up the areas of all the bars of a relative frequency histogram gives a total area (i.e. total proportion) of one. Integration is itself a sort of continuous version of taking a sum.

R code for plots

require(ggplot2)
require(scales)
require(gridExtra)
# Code for the PDF plots with bars underneath could be easily readapted

# Relative frequency histograms
x.df <- data.frame(km=c(rep(12.5, 1), rep(17.5, 2), rep(22.5, 5), rep(27.5, 2)))
x.df$metres <- x.df$km * 1000

km.plot <- ggplot(x.df, aes(x=km, y=..density..)) +
  stat_bin(origin=10, binwidth=5, fill="steelblue", colour="black") +
  xlab("Distance in km") + ylab("Relative frequency density per km") +
  scale_y_continuous(minor_breaks = seq(0, 0.1, by=0.005))

metres.plot <- ggplot(x.df, aes(x=metres, y=..density..)) +
  stat_bin(origin=10000, binwidth=5000, fill="steelblue", colour="black") +
  xlab("Distance in metres") + ylab("Relative frequency density per metre") +
  scale_x_continuous(labels = comma) +
  scale_y_continuous(minor_breaks = seq(0, 0.0001, by=0.000005), labels=comma)

grid.arrange(km.plot, metres.plot, ncol=2)
x11()

# Probability density functions
x.df <- data.frame(x=seq(0, 1, by=0.001))
cutoffs <- seq(0.2, 0.5, by=0.1) # for bars
barHeights <- c(0, dbeta(cutoffs[1:(length(cutoffs)-1)], 2, 2), 0) # uses left of bar

x.df$pdf <- dbeta(x.df$x, 2, 2)
x.df$bar <-  findInterval(x.df$x, cutoffs) + 1 # start at 1, first plotted bar is 2
x.df$barHeight <- barHeights[x.df$bar]

x.df$lastBar <- ifelse(x.df$bar == max(x.df$bar)-1, 1, 0) # last plotted bar only
x.df$lastBarHeight <- ifelse(x.df$lastBar == 1, x.df$barHeight, 0)
x.df$integral <- ifelse(x.df$bar %in% 2:(max(x.df$bar)-1), 1, 0) # all plotted bars
x.df$integralHeight <- ifelse(x.df$integral == 1, x.df$pdf, 0)

cutoffsNarrow <- seq(0.2, 0.5, by=0.025) # for the narrow bars
barHeightsNarrow <- c(0, dbeta(cutoffsNarrow[1:(length(cutoffsNarrow)-1)], 2, 2), 0) # uses left of bar
x.df$barNarrow <-  findInterval(x.df$x, cutoffsNarrow) + 1 # start at 1, first plotted bar is 2
x.df$barHeightNarrow <- barHeightsNarrow[x.df$barNarrow]

pdf.plot <- ggplot(x.df, aes(x=x, y=pdf)) +
  geom_area(fill="lightsteelblue", colour="black", size=.8) +
  ylab("probability density") +
  theme(panel.grid = element_blank(),
  axis.text.x = element_text(colour="black", size=16))

pdf.lastBar.plot <- pdf.plot +
  scale_x_continuous(breaks=tail(cutoffs, 2), labels=expression(x, x+delta*x)) +
  geom_area(aes(x=x, y=lastBarHeight, group=lastBar), fill="steelblue", colour="black", size=.8) +
  annotate("text", x=0.73, y=0.22, size=6, label=paste("P(paste(x<=X)<=x+delta*x)%~~%f(x)*delta*x"), parse=TRUE)

pdf.bars.plot <- pdf.plot +
  scale_x_continuous(breaks=cutoffs[c(1, length(cutoffs))], labels=c("a", "b")) +
  geom_area(aes(x=x, y=barHeight, group=bar), fill="steelblue", colour="black", size=.8) +
  annotate("text", x=0.73, y=0.22, size=6, label=paste("P(paste(a<=X)<=b)%~~%sum(f(x)*delta*x)"), parse=TRUE)

pdf.barsNarrow.plot <- pdf.plot +
  scale_x_continuous(breaks=cutoffsNarrow[c(1, length(cutoffsNarrow))], labels=c("a", "b")) +
  geom_area(aes(x=x, y=barHeightNarrow, group=barNarrow), fill="steelblue", colour="black", size=.8) +
  annotate("text", x=0.73, y=0.22, size=6, label=paste("P(paste(a<=X)<=b)%~~%sum(f(x)*delta*x)"), parse=TRUE)

pdf.integral.plot <- pdf.plot +
  scale_x_continuous(breaks=cutoffs[c(1, length(cutoffs))], labels=c("a", "b")) +
  geom_area(aes(x=x, y=integralHeight, group=integral), fill="steelblue", colour="black", size=.8) +
  annotate("text", x=0.73, y=0.22, size=6, label=paste("P(paste(a<=X)<=b)==integral(f(x)*dx,a,b)"), parse=TRUE)

grid.arrange(pdf.lastBar.plot, pdf.bars.plot, pdf.barsNarrow.plot, pdf.integral.plot, ncol=2)

Answer 3

You already got two answers, with an excellent one by Silverfish, however I feel that an illustration could be useful in here since you asked about geometry and "imagining" yourself those functions.

Lets start with a simple example of Bernoulli distribution:

$$ f(x) = \begin{cases} p & \text{if }x=1, \\[6pt] 1-p & \text {if }x=0.\end{cases} $$

Since the values are discrete there is no "curve" but only two points, however the idea is similar: if you want to know total probability (area under the curve) you have to sum up probabilities of both possible outcomes:

$$p + (1 - p) = 1$$

There is only $p$ and $1-p$ in this equation since we have only two possible point outcomes with a given probabilities.

The same would be with Poisson distribution that is also a discrete probability distribution. There are more than two values, so you can imagine that there is a line that connects the points, however to compute the total probability you would have to sum up all the probabilities of $x$'s. Poisson distribution is often used to describe count data, so you can think of it as each $x$ is a number of certain events and $f(x)$ is a probability of this outcome. You could imagine that each point on the plot below is actually a height of a stack made of some outcomes: $x_1$ is a stack of all the "$x_1$" outcomes that you observed etc. The total "area under the curve" would be here all the stacks summed up (or a meta-stack of all the outcomes) but since we do not sum up numbers of occurrences but rather probabilities, they sum up to $1$. So you should not think of it as sum of counts $\sum \#\{x_i\}=N$, but rather as sum of probabilities: $\sum \#\{x_i\}/N=1$ where $N$ is a total number of all possible outcomes.

Now let's consider a normal distribution that actually is a continuous distribution - so we don't have "points" since the values of $x$ are on continuous scale, i.e. there is infinitely many values of $x$. So if there were points you couldn't see them no matter how much would you "zoom in", since there always could be some an infinite number of smaller points between any given points. Because of that here we actually have a curve - you can imagine that it is made of infinitely many "points". You could ask yourself: how to compute a sum of infinite number of probabilities..? On the plot below red curve is a normal PDF and the black boxes is histogram of some values drawn from the distribution. So histogram plot has simplified our distribution to the finite number of "boxes" with a certain width and if you summed up the heights of the boxes multiplied by their width you would end up with an area under the curve - or area of all the boxes. We use areas rather points in here since each box is a summary of an infinite number of "points" that were packed up in the box.

So to get total area we take the heights (i.e. $f(x)$) and widths (e.g. first box has width: $-2.5 - -3 = 0.5$, the same as all the other boxes). In the actual figure plotted the heights of the boxes are:

0.010 0.028 0.094 0.198 0.260 0.400 0.404 0.292 0.166 0.092 0.044 0.010 0.002

if you sum them up multiplying each by $0.5$ (width), they will sum up to $1$. In here you cannot count anything since there is infinitely many possible points that form the curve. On another hand, since we are talking about probabilities, the probability of all the possible outcomes has to be $1$.

In this case we use "probability per unit" and the unit can have any width of your choice. Consider "all possible outcomes" on the continuous scale as a line that could be divided into the parts, and each part could be divided into some smaller parts up to infinitely small ones. The total probability of this line is $1$. If it would be flat than you could imagine that it's total length is $1$ and by dividing it you get probabilities of the parts. If the line is not flat, the probability per part is described by function $f(x)$. So the units actually doesn't matter since there is infinite number of possible "points" it is probability per unit, where unit is always the same: a fraction of "total" length.

This approach illustrates in a simplified way a little bit more complicated issue - taking integrals. In continuous case you use integrals for calculating the area under the curve. Integral of the area of the curve between points $a$ and $b$ ($-3$ and $3$ on out plot) is:

$$\int_a^b \! f(x)\,dx$$

where $f(x)$ is height and $dx$ is width and you could think of $\int$ as $\sum$ for continuous variables. For learning more on integrals and calculus you could check the Khan Academy lectures.

You asked also about the "flat" (uniform) distribution:

First notice that this is not a valid uniform distribution since it should have parameters such that $-\infty < a < b < \infty$, so to integrate to $1$. If you think of it, it is continuous and since it is flat, it is some kind of box with a width from $-\infty$ to $\infty$. If you wanted to calculate area of such box, you would be multiplying the height by width. Unfortunately, while the width is infinitely wide, for it to integrate to $1$ the height would have to be some $\varepsilon$ that is enormously small... So this is a complicated case and you could imagine it rather in abstract terms. Notice that, as Ilmari Karonen noticed in the comment, this is rather an abstract idea that is not really possible in practice (see the comment below). If using such distribution as a prior, it would be an improper prior.

Notice that in the continuous case probability density function gives you density estimates rather then probabilities, so heights (or their sum) could exceed $1$ (see here for more).

Answer 4

The following key idea was mentioned in a comment, but not in an existing answer...

One way of intuiting about the properties of a PDF is to consider that the PDF and the CDF are related by integration (calculus) -- and that the CDF has a monotonic output representing a probability value between 0 and 1.

The unitless integrated total area under the PDF curve is not affected by X-axis units.

To put it simply...

Area = Width x Height

If the X-axis gets larger, numerically, due to a change in units, then the Y-axis must become smaller by a corresponding linear factor.