Can a probability distribution value exceeding 1 be OK?

Question

On the Wikipedia page about naive Bayes classifiers, there is this line:

$p(\mathrm{height}|\mathrm{male}) = 1.5789$ (A probability distribution over 1 is OK. It is the area under the bell curve that is equal to 1.)

How can a value $>1$ be OK? I thought all probability values were expressed in the range $0 \leq p \leq 1$. Furthermore, given that it is possible to have such a value, how is that value obtained in the example shown on the page?

Answer 1

That Wiki page is abusing language by referring to this number as a probability. You are correct that it is not. It is actually a probability per foot. Specifically, the value of 1.5789 (for a height of 6 feet) implies that the probability of a height between, say, 5.99 and 6.01 feet is close to the following unitless value:

$$1.5789\, [1/\text{foot}] \times (6.01 - 5.99)\, [\text{feet}] = 0.0316$$

This value must not exceed 1, as you know. (The small range of heights (0.02 in this example) is a crucial part of the probability apparatus. It is the "differential" of height, which I will abbreviate $d(\text{height})$.) Probabilities per unit of something are called densities by analogy to other densities, like mass per unit volume.

Bona fide probability densities can have arbitrarily large values, even infinite ones.

This example shows the probability density function for a Gamma distribution (with shape parameter of $3/2$ and scale of $1/5$). Because most of the density is less than $1$, the curve has to rise higher than $1$ in order to have a total area of $1$ as required for all probability distributions.

This density (for a beta distribution with parameters $1/2, 1/10$) becomes infinite at $0$ and at $1$. The total area still is finite (and equals $1$)!

---

The value of 1.5789 /foot is obtained in that example by estimating that the heights of males have a normal distribution with mean 5.855 feet and variance 3.50e-2 square feet. (This can be found in a previous table.) The square root of that variance is the standard deviation, 0.18717 feet. We re-express 6 feet as the number of SDs from the mean:

$$z = (6 - 5.855) / 0.18717 = 0.7747$$

The division by the standard deviation produces a relation

$$dz = d(\text{height})/0.18717$$

The Normal probability density, by definition, equals

$$\frac{1}{\sqrt{2 \pi}}\exp(-z^2/2)dz = 0.29544\ d(\text{height}) / 0.18717 = 1.5789\ d(\text{height}).$$

(Actually, I cheated: I simply asked Excel to compute NORMDIST(6, 5.855, 0.18717, FALSE). But then I really did check it against the formula, just to be sure.) When we strip the essential differential $d(\text{height})$ from the formula only the number $1.5789$ remains, like the Cheshire Cat's smile. We, the readers, need to understand that the number has to be multiplied by a small difference in heights in order to produce a probability.

Answer 2

This is a common mistake from not understanding the difference between probability mass functions, where the variable is discrete, and probability density functions, where the variable is continuous. See What is a probability distribution:

continuous probability functions are defined for an infinite number of points over a continuous interval, the probability at a single point is always zero. Probabilities are measured over intervals, not single points. That is, the area under the curve between two distinct points defines the probability for that interval. This means that the height of the probability function can in fact be greater than one. The property that the integral must equal one is equivalent to the property for discrete distributions that the sum of all the probabilities must equal one.

Answer 3

I think that a continuous uniform distribution over an interval $[a,b]$ provides a straightforward example for this question: In a continuous uniform distribution the density in each point is the same at each point (uniform distribution). Moreover, because the area below the rectangle must be one (just as the area below the normal curve must be one) that density value must be $1/(b-a)$ because any rectangle with base $b-a$ and area $1$ must have height $1/(b-a)$ .

So the value for the uniform density on the interval $[0,0.5]$ is $1/(0.5-0)=2$, on the interval $[0,0.1]$ it is $10$, ...

Answer 4

I don't know whether the Wikipedia article has been edited subsequent to the initial posts in this thread, but it now says "Note that a value greater than 1 is OK here – it is a probability density rather than a probability, because height is a continuous variable.", and at least in this immediate context, P is used for probability and p is used for probability density. Yes, very sloppy since the article uses p in some places to mean probability, and in other places as probability density.

Back to the original question "Can a probability distribution value exceeding 1 be OK?" No, but I've seen it done (see my last paragraph below).

Here's how to interpret a probability > 1. First of all, note that people can and do give a 150% effort, as we often hear in sports and sometimes work https://www.youtube.com/watch?v=br_vSdAOHQQ . If you're sure something will happen, that's a probability of 1. A probability of 1.5 could be interpreted as you're 150% sure the event will happen - kind of like giving a 150% effort.

And if you can have a probability > 1, I suppose you can have a probability < 0. Negative probabilities can be interpreted as follows. A probability of 0.001 means there's almost no chance of the event happening. Probability = 0 means "no way". A negative probability, such as -1.2, corresponds to "You gots to be kidding".

When I was a wee lad just out of school 3 decades ago, I witnessed an event more astounding than breaking the sound barrier in aviation, namely, breaking the unity barrier in probability. An analyst with a Ph.D. in Physics had spent 2 years full-time (probably giving 150%) developing a model for calculating the probability of detecting object X, at the end of which his model and analysis successfully completed peer review by several scientists and engineers closely affiliated to the U.S. government. I won't tell you what object X is, but object X, and the probability of detecting it, was and still is of considerable interest to the U.S. government. The model included a formula for $P_y$ = Prob(event y happens). $P_y$ and some other terms all combined into the final formula, which was Prob(object X is detected). Indeed, computed values of Prob(object X is detected) were within the range of [0,1], as is "traditional" in probability in the Kolmogorov tradition. $P_y$ in its original form was always in [0,1] and involved "garden-variety" transcendental functions which were available in standard Fortran or any scientific calculator. However, for a reason known only but to the analyst and God (perhaps because he had seen it done in his Physics classes and books, but did not know that he was shown the few cases where it works, not the many more where it does not, and this guy's name and scientific/mathematical judgment did not happen to be that of Dirac), he chose to take a two term Taylor expansion of $P_y$ (and ignore the remainder term), which will henceforth be referred to as $P_y$. It was this two term Taylor expansion of $P_y$ which was inserted into the final expression for Prob(object X is detected). What he did not realize, until I pointed it out to him, was that $P_y$ was equal to approximately 1.2 using his base case values for all parameters. Indeed it was possible for $P_y$ to go up to about 1.8. And that's how the unity barrier was broken in probability. But the guy didn't know he had accomplished this pioneering feat until I pointed it out to him, having just performed quick calculations on a battery-powered credit card size Casio scientific calculator in a darkened conference room (couldn't have done it with a solar-powered calculator). That would be kind of like Chuck Yeager going out for a Sunday spin in his plane, and only being informed months later that he had broken the sound barrier.

Answer 5

When random variable $X$ is continuous and its probability density function is $f(x)$, $f(x)dx$ is a probability, but $f(x)$ is not a probability and can be larger than one. The reported $f(\mbox{height}|\mbox{male})$ is not a probability, but $f(\mbox{height}|\mbox{male})d\mbox{height}$ is.

In other words, for a continuous random variable $X$, $P(X\in[x,x+dx))=f(x)dx$, $P(X\in[a,b])=\int_{a}^{b}f(x)dx$, and $P(X = x)=P(X \in [x,x])=0$. The same goes for conditional probabilities.

Answer 6

The point value at a particular parameter value of a probability density plot would be a likelihood, right? If so, then the statement might be corrected by simply changing P(height|male) to L(height|male).