20
$\begingroup$

I know that for any continuous variable $P[X=x]=0$.

But I can't visualize that if $P[X=x]=0$, there is an infinite number of possible $x$'s. And also why do their probabilities get infinitely small ?

$\endgroup$
8
  • $\begingroup$ possible duplicate of Conditional probability of continuous variable $\endgroup$
    – Xi'an
    Mar 21, 2015 at 15:37
  • 3
    $\begingroup$ There are already two votes to close this question as duplicate. I don't agree. This is a pretty basic topic, one of those that will probably re-appear in the future, so it would be good if it had a direct and high quality answer, so we could refer to it in the future. The link provided by @Xi'an may be threated as duplicate but is also quite specific and hard to find via search. The link also does not provide an exhaustive answer, while this threat seems to converge to such. I think it should be left open as a future reference. $\endgroup$
    – Tim
    Mar 21, 2015 at 16:51
  • 1
    $\begingroup$ It might help to consider the inverse of this situation. Let $X$ be any random variable and let $\epsilon$ be any positive real number. There can be only a finite number of $\omega$ for which $\Pr(X=\omega)\ge\epsilon$, for otherwise--by adding up all these probabilities over disjoint events--you would conclude that the total probability is at least $\epsilon+\epsilon+\cdots$, which eventually exceeds $1$. (This is the Archimedean property of real numbers.) This reasoning uses only three axioms: probabilities of disjoint events add, total probability is $1$, and the Archimedean axiom. $\endgroup$
    – whuber
    Mar 21, 2015 at 17:07
  • 1
    $\begingroup$ @Tim Thank you, but I posted this thought as a comment, rather than an answer, because it's incomplete: I haven't figured out an elementary way to explain what happens in the limit as $\epsilon\to 0$. It seems to require some knowledge of cardinalities of infinite sets. $\endgroup$
    – whuber
    Mar 21, 2015 at 17:32
  • 4
    $\begingroup$ @Xi'an I agree, but the thread you proposed is not a sufficiently close duplicate. This is a difficult thing to search for. Are you perhaps aware of other threads that duplicate this question? $\endgroup$
    – whuber
    Mar 21, 2015 at 18:23

5 Answers 5

19
$\begingroup$

Probabilities are models for the relative frequencies of observations. If an event $A$ is observed to have occurred $N_A$ times on $N$ trials, then its relative frequency is $$\text{relative frequency of }(A) = \frac{N_A}{N}$$ and it is generally believed that the numerical value of the above ratio is a close approximation to $P(A)$ when $N$ is "large" where what is meant by "large" is best left to the imagination (and credulity) of the reader.

Now, it has been observed that if our model of $X$ is that of a continuous random variable, then the samples of $X$ $\{x_1, x_2, \ldots, x_N\}$ are $N$ distinct numbers. Thus, the relative frequency of a specific number $x$ (or, more pedantically, the event $\{X = x\}$) is either $\frac 1N$ if one of the $x_i$ has value $x$, or $\frac 0N$ if all the $x_i$ are different from $x$. If a more skeptical reader collects an additional $N$ samples, the relative frequency of the event $\{X=x\}$ is either $\frac{1}{2N}$ or continues to enjoy the value $\frac 0N$. Thus, one is tempted to guess that $P\{X = x\}$ should be assigned the value $0$ since that is a good approximation to the observed relative frequency.

Note: the above explanation is (usually) satisfactory to engineers and others interested in the application of probability and statistics (i.e. those who believe that the axioms of probability were chosen so as to make the theory a good model of reality), but totally unsatisfactory to many others. It is also possible to approach your question from a purely mathematical or statistical perspective and prove that $P\{X = x\}$ must have value $0$ whenever $X$ is a continuous random variable via logical deductions from the axioms of probability, and without any reference to relative frequency or physical observations etc.

$\endgroup$
6
  • 1
    $\begingroup$ +1 "Note: the above explanation is... satisfactory to... those who believe that the axioms of probability were chosen so as to make the theory a good model of reality), but totally unsatisfactory...", in the internet's preferred phrasing, lol. $\endgroup$ Mar 21, 2015 at 16:30
  • 2
    $\begingroup$ I don't understand what do you mean by it has been observed that if $X$ is continuous, then .... How can we observe that ? $\endgroup$ Mar 21, 2015 at 17:11
  • 3
    $\begingroup$ @StéphaneLaurent That sentence is a little complicated, so it bears re-reading. Stripped of some parenthetical remarks, it says "it has been observed that ... the samples ... are $N$ distinct numbers." In other words, when one assumes that $X$ has a continuous distribution, then (almost surely) there will be no duplicates in any finite iid sample of $X$. That can be mathematically proven: it's not a mere observation. $\endgroup$
    – whuber
    Mar 21, 2015 at 21:12
  • 2
    $\begingroup$ @StéphaneLaurent I think Dilip's remarks are being made in a different spirit than that. This answer is not an effort to provide a mathematically rigorous demonstration, but to provide some intuition and motivation for a fact that puzzles the OP. I am intrigued by this approach because it has such potential to bridge the gap between the discrete probability theory traditionally taught to beginners and the richer general theory of probability based on measure theory. $\endgroup$
    – whuber
    Mar 21, 2015 at 21:25
  • 2
    $\begingroup$ @whuber I understand the spirit, but at first glance I was not convinced that the no-ties property is more intuitive than the zero-probability property. For $N=2$ this is really the same thing: "$x_2 \text{ is never } x_1$" $\iff$ $\Pr(X_2=x_1)=0$. $\endgroup$ Mar 21, 2015 at 21:53
17
$\begingroup$

Let $(\Omega,\mathscr{F},P)$ be the underlying probability space. We say that a measurable function $X:\Omega\to\mathbb{R}$ is an absolutely continuous random variable if the probability measure $\mu_X$ over $(\mathbb{R},\mathscr{B})$ defined by $\mu_X(B)=P\{X\in B\}$, known as the distribution of $X$, is dominated by Lebesgue measure $\lambda$, in the sense that for every Borel set $B$, if $\lambda(B)=0$, then $\mu_X(B)=0$. In this case, the Radon-Nikodym theorem tells us that there is a measurable $f_X:\mathbb{R}\to\mathbb{R}$, defined up to almost everywhere equivalence, such that $\mu_X(B)=\int_B f(x)\,d\lambda(x)$. Let $B=\{x_1,x_2,\dots\}$ be a countable subset of $\mathbb{R}$. Since $\lambda$ is countably additive, $\lambda(B)=\lambda\left(\cup_{i\geq 1}\{x_i\}\right)=\sum_{i\geq 1}\lambda(\{x_i\})$. But $$ \lambda(\{x_i\}) = \lambda\left(\cap_{k\geq 1}[x_i,x_i+1/k)\right) \leq \lambda\left([x_i,x_i+1/n)\right) = \frac{1}{n} \, ,\qquad (*) $$ for every $n\geq 1$. Due to the Archimedean property of the real numbers, since $\lambda(\{x_i\})\geq 0$, the inequality $(*)$ holds for every $n\geq 1$ if and only if $\lambda(\{x_i\})=0$, entailing that $\lambda(B)=0$. From the assumed absolute continuity of $X$ it follows that $\mu_X(B)=P\{X\in B\}=0$.

$\endgroup$
4
  • 1
    $\begingroup$ Continuous random variable doesn't need to be absolutely continuous (it could have no density.) $\endgroup$
    – Zhanxiong
    Jul 24, 2015 at 13:49
  • 3
    $\begingroup$ Baloney. "Continuous random variable" is an informal name for "a random variable which is absolutely continuous with respect to Lebesgue measure". Hence, Radon-Nikodym guarantees that a density exists. A random variable with a singular distribution (e.g. Cantor) is a different thing. You're misleading potential students with your bogus comment. $\endgroup$
    – Zen
    Jul 24, 2015 at 14:04
  • $\begingroup$ When you criticized someone, please show the citation you referred to. Which probability text book said that "Continuous random variable" is an informal name for "a random variable which is absolutely continuous with respect to Lebesgue measure"? In addition, this problem can be solved without requiring $X$ has a density, see my proof below. $\endgroup$
    – Zhanxiong
    Jul 24, 2015 at 14:13
  • $\begingroup$ Wikipedia disagrees with you, @Solitary: "A continuous probability distribution is a probability distribution that has a probability density function. Mathematicians also call such a distribution absolutely continuous [...]". $\endgroup$
    – amoeba
    Jan 18, 2016 at 11:41
6
$\begingroup$

$X$ is a continuous random variable means its distribution function $F$ is continuous. This is the only condition we have but from which we can derive that $P(X = x) = 0$.

In fact, by continuity of $F$, we have $F(x) = F(x-)$ for every $x \in \mathbb{R}^1$, therefore: $$P(X = x) = P(X \leq x) - P(X < x) = F(x) - F(x-) = 0.$$

$\endgroup$
4
  • 1
    $\begingroup$ If the distribution of a r.v. $X$ is Cantor, then its distribution function is continuous, but $X$ is a singular random variable; it's not a continuous random variable. $\endgroup$
    – Zen
    Jul 24, 2015 at 14:22
  • 2
    $\begingroup$ My friend, this actually can be a counterexample to your own answer, not mine. Since the existence of such Singular continuous r.v., it is necessary to distinguish absolute continuous r.v. and singular continuous r.v., although their distribution functions are all continuous. To equalize continuous r.v. and absolute continuous r.v. is ambiguous. $\endgroup$
    – Zhanxiong
    Jul 24, 2015 at 14:54
  • 1
    $\begingroup$ It isn't, but you won't hear, my friend. $\endgroup$
    – Zen
    Jul 24, 2015 at 21:03
  • 1
    $\begingroup$ By the way, you're actually "proving" that if $P(X=x)=0$ for every $x$, then $P(X=x)=0$ for every $x$. $\endgroup$
    – Zen
    Jul 24, 2015 at 21:06
2
$\begingroup$

This is really a question about probabilities.

It could be rephrased something like

Suppose there are an infinite number of disjoint events. Why must their probabilities eventually become arbitrarily small?

Well, if they didn't become small, there would be a positive number $\epsilon$ smaller than all those probabilities. One axiom of probability implies the probability of a union of a finite number of distinct events is the sum of their probabilities. Since "infinite" means there exist finite subsets of any size $n,$ we at least know that for any whole number $n$ there is a set of $n$ distinct events of probability $\epsilon$ or greater. Taking $n \gt 1/\epsilon$ (guaranteed by the Archimedean property of real numbers) we would deduce the existence of a collection of disjoint events whose probability exceeds $n\epsilon\gt 1,$ an obvious impossibility. This completes the proof: we are obliged to conclude the original assumption is false: namely, the probabilities do become arbitrarily small.

BTW, it should be no mystery why an infinite number of disjoint events can all have zero probability. Take the simplest kind of random variable: the constant $X=0.$ For any nonzero $x,$ obviously $\Pr(X=x) = 0.$ There are a lot of such $x$! The problem really is,

how can it be possible that $\Pr(X=x)=0$ for all possible values $x$ without forcing all probabilities to be zero?

The formal mathematical answer is buried in the probability axioms: they do not state, nor even imply, that the probability of an event must be the sum of the probabilities of all its elements. This is the subtlety in the restriction to countable unions and countable sums. The real numbers are not countable.

But then how do you visualize such a thing? The cumulative distribution function $F_X$ of any random variable $X$ is a useful tool. By definition, for any number $x,$ $F_X(x)$ is the chance that $X \le x,$ written $\Pr(X\le x).$ We can (and usually do) draw a graph of $F_X.$ The probability axioms imply its heights must lie between $0$ and $1$ (they are all probabilities, after all) and that the graph never decreases as you increase $x.$ (This latter is proven by noting that for $y\gt x,$ $F(y)-F(x) = \Pr(y \lt X \le x) \ge 0.$)

Using the CDF, then, we find that the probability of $X=x$ is the amount by which the graph of $F$ rises at $x$ as you move left to right. In any continuous such graph, all such rises are zero (that's essentially the definition of "continuous"). Nevertheless, despite rising by an amount $0$ at every point $x,$ somehow the graph manages to increase its elevation from $0$ to $1$ throughout its domain. See Wikipedia (at "CDF") for some examples. Pondering this visually evident fact might help you appreciate these issues better. For a more insight, study Zeno's Paradoxes.

$\endgroup$
0
$\begingroup$

This question is very simple. PDF is the density, then to get the probability you need to multiply it by the width of the region. So, when you get a smaller region around the point of interest $x$ the height of the density doesn't change, hence the probability is smaller and smaller while you squeeze the area around your point until it becomes exactly zero.

The difficult question is the reverse one, like this: https://stats.stackexchange.com/a/273407/36041

$\endgroup$
1
  • 1
    $\begingroup$ Although this is helpful, it would be good to note that not all continuous variables have densities: only absolutely continuous variables have them. $\endgroup$
    – whuber
    Mar 21 at 20:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.