What is meant by the probability of a sample having a value of $x$ is $ng(x)$?

Question

Reading from Wikipedia:

The probability of one sample having a value of $x$ is $n g(x)$.

Assuming that the notation is consistent throughout the page, I would take $g$ to either be the probability mass function or the probability density function... I have some confusions with this statement that I want to clear up.

If $g$ is meant to be a density function, then there exists a density $g$ for a normal $\mathcal{N}(\mu, \sigma)$ where $\mu$ and $\sigma$ chosen sufficiently to guarantee $1 < ng(x)$ for some given $x$ and sample size $n$. So that cannot be a probability if it can exceed unity.

If $g$ is meant to be a probability mass function, then there exists a sufficiently-large sample size $n$ and $0 < g(x)$ such that $1 < n g(x)$. Thus $n g(x)$ is not in general a probability either.

I'm unsure what I'm missing here, but this premise doesn't make sense to me in terms of Kolmogorov's axioms because they imply that $\Pr [x \in A] \leq 1$ for any $A$ in the event space.

Answer 1

With a generous interpretation we can make sense of this quasi-argument.

Consider an iid sequence of absolutely continuous random variables $\mathbf X = X_1,X_2,\ldots, X_n$ with density $g$ and cumulative distribution function $G.$ For any number $x$ and positive real number $t;$ and for any ordered pair $1\le i \lt j \le n,;$ consider the intersection of the events $X_i=x,$ $X_j=x+t,$ and $x\lt X_k\lt x+t$ for all $k\ne i,j.$ Let's call this intersection $\mathscr{E}_{i,j}(x,t).$ Because for any $X_k$ we can obtain the probability of the interval $(x,x+t)$ as

$$\Pr(x\lt X_k\lt x+t) = G(x+t)-G(x),$$

independence implies the density of this event is obtained by multiplying the $n$ separate probabilities:

$$\Pr(\mathscr{E}_{i,j}(x,t))\mathrm dx\mathrm dt = g(x)g(x+t)(G(x+t)-G(x))^{n-2}\,\mathrm dx\mathrm dt.\tag{*}$$

There are $n(n-1)$ such events corresponding to all choices of the distinct pair $(i,j)$ (since, echoing the words in Wikipedia, the are $n$ choices for $i$ and then $n-1$ more choices for $j$ independently of $i$).

Here's the punch line: the event "the range is $t$ and the minimum is $x$" is the union of all such events. In mathematical symbols,

$$(\operatorname{Range}(\mathbf X) = t)\,\cap\, (\min(\mathbf X) = x)\ = \bigcup_{i\ne j}\mathscr{E}_{i,j}(x,t).$$

Because all the events on the right are disjoint (each is determined by which of the $X_k$ equal the two extremes), their probabilities add, giving

$$\Pr((\operatorname{Range}(\mathbf X) = t)\,\cup\, (\min(\mathbf X) = x))\,\mathrm dx\mathrm dt = \sum_{i,j} \Pr(\mathscr{E}_{i,j}(x,t))\,\mathrm dx\mathrm dt$$

Since by $(*)$ all the terms on the right hand side are equal, we obtain $n(n-1)$ times the right hand side of $(*),$ equal to

$$n(n-1)g(x)g(x+t)(G(x+t)-G(x))^{n-2}\,\mathrm dx\mathrm dt,$$

as claimed. Because it was obtained using (a) the definition of independence and (b) a basic probability axiom, it's a bona fide (but infinitesimal!) probability.

I hope that in light of this (slightly more rigorous) argument, the idea lurking behind the Wikipedia gibberish is revealed.

Answer 2

Looks like the Wikipedia page is a little (very?) loose with definitions. Given the integral in the section from which you pulled the quote, $g(x)$ is almost certainly referring to a probability density function, although I wouldn't be surprised if similar formulae (e.g. a summation instead of an integral) would apply for a probability mass function as well.

The probability that any continuous random variable equals any particular value is 0, not the density. So the statement you quote of "The probability of one sample having a value of $x$ is $ng(x)$" is wrong. If you are hung up on $ng(x)$ being greater than 1, remember that any density function can have values greater than 1. One example being the Uniform(0, .5) distribution, whose pmf is equal to 2 between 0 and .5 and equal to 0 everywhere else.

I think a better approach to explaining that section would be to go from the inside out. First state that in order for the range to be $t$, one sample point must equal $x$, another must equal $x+t$, and then all other points must be between $x$ and $x + t$. Then give a combinatorics explanation that the $x$ and $x+t$ values could fall anywhere in the sample, so we have to multiply by $n(n-1)$. Then finally note that thus far we have only calculated $t | x$ so we need to integrate out $x$ to get the unconditional distribution.

Answer 3

The formula relates to continuous distributions. For continuous distributions it makes no sense to speak about "the probability of one sample having a value of $x$" because the probability is zero (as in $P[X=x]=0$ when $X$ is a continuous variable).

However, we can consider a hand waving argument that we consider a discretized distribution with small bins of size $\delta$ and probability $g(x)\delta$ and they have a very small probability such that no bin is occupied more than once.

Then we consider the distribution of $n$ samples with

• 1 specific sample into the bin $x$ with probability $g(x) \delta$

  and any of the $n$ samples into the bin $x$ with probability $1-(1-g(x)\delta)^n \approx ng(x)\delta$

• 1 specific sample into the bin $x+t$ with probability $g(x+t) \delta$

  and any of the remaining $n-1$ samples into the bin $x+t$ with probability $1-(1-g(x+t)\delta)^{n-1} \approx (n-1)g(x+t)\delta$

• n-2 of the remaining samples in between bins $x$ and $x+t$ with probability $(G(x+t)-G(t))^{n-2}$

the probability that the minimum is in bin $x$ and the maximum in bin $x+t$ is the product

$$P( \min = x \land \max = x+t) = n(n-1) g(x)g(x+t) (G(x+t)-G(t))^{n-2} \cdot \delta^2$$

the term $n(n-1)$ arrises because of the multiple ways that the samples can be distributed among the three cases, minimum, maximum ornin between. For the minimum we can pick $n$ samples , and for the maximum we can pick $n-1$ of the remaining samples.

If $g$ is meant to be a probability mass function, then there exists a sufficiently-large sample size $n$ and $0 < g(x)$ such that $1 < n g(x)$. Thus $n g(x)$ is not in general a probability either

The real probability should be computed as

$$1-(1-g(x))^n \approx ng(x)$$

and it is approximately equal to $ng(x)$ if the probabilities are small.