What is a probability distribution? [duplicate]

Question

This is a very basic question, and maybe a silly one, but I'm struggling to understand the actual definition of a probability distribution.

In Wasserman's "All of Statistics", for example, he says that "if $\Omega$ is finite and if each outcome is equally likely, then $\mathbb{P}(A) = |A| / |\Omega|$, which is called the uniform probability distribution."

This suggest to me that a probability distribution refers specifically to the probability measure chosen, i.e. a probability distribution is the measure associated with a probability space. Yet on Wikipedia and elsewhere online, a probability distribution is only defined within the context of some random variable, e.g. "A probability distribution is a function that describes the possible values of a random variable and their associated probabilities". So my question is, is a probability distribution the measure on a given space, or is it the pdf/pmf of a given random variable? When we say we "draw a random variable" from a probability distribution, does that mean we construct a random variable from a given probability space with that probability measure, or that our random variable has that pdf/pmf? Is this a meaningful distinction?

Answer 1

"A probability distribution is a specification of the stochastic structure of a random variable." (Gentle, Matrix Algebra, Chapt. 9)

That said, yes, you are correct to assume that in terms of distances in probability theory, the measure $P$ on $A$ is what is called probability measure, or probability distribution law, or probability distribution. (Here $A$ is the set of all measurable subsets of $\Omega$ where $\Omega$ itself is the sample space.) (See, Deza & Deza, Encyclopedia of Distances, Chapt. 14)

Therefore I think the answer to the question "what is a probability distribution" depends on the context of that the phrase is used. For most applied purposes, e.g. "drawing a random variable" from a probability distribution that we associate with our data, using the concept of the variable having a particular PMF/PDF is perfectly adequate; nobody explicitly defines a probability space but just alludes indirectly to it by assuming a particular probability distribution. This is actually what is done computationally too; during random number generation in most cases, either through the inverse CDF of a distribution or through some composite scheme (e.g. rejection sampling) we enforce a particular PMF/PDF in our sample. Mathematically though, yes, we do need the concept of a measure $P$ on a given space to define a random variable.

There is an excellent thread on why do we need σ -algebras to define probability spaces? which questions more the mechanism around a probability space. To that extent, Math.SE also has some very relevant topics on the difference between density and distribution [in formal mathematical terms] and the difference between “probability density function” and “probability distribution function”.

Answer 2

A probability distribution $\mathbb{P}$ is defined as a positive measure on a measurable space ${\cal X}$ with $\sigma$-algebra ${\cal A}$ and total mass one, $\mathbb{P}({\cal X})=1$. It is naturally associated with a random variable in that each random variable has a probability distribution and, given a probability distribution, one can always construct a random variable with this probability distribution. To quote Terry Tao's 254A, Notes 0: A review of probability theory, a random variable is a measurable transform from a sample space $\Omega$ endowed with a probability distribution $(\Omega,{\cal B},\mu)$ with $\mu(\Omega)=1$, onto a set ${\cal X}$:

Definition 3 (Random variable) Let $R = (R,{\mathcal R})$ be a measurable space (i.e. a set $R$, equipped with a $\sigma$-algebra of subsets of ${\mathcal R}$). A random variable taking values in $R$ (or an $R$-valued random variable) is a measurable map $X$ from the sample space to $R$, i.e. a function$$X: \Omega \rightarrow R$$such that $X^{-1}(S)$ is an event for every $S \in {\mathcal R}$.

(although I do not deem the measurability of $R$ necessary to the definition, since ${\mathcal R}=X(\mathcal B)$ is automatically defined as the transformed $\sigma$-algebra). And

Lemma 4 (Creating a random variable with a specified distribution) Let $\mu$ be a probability measure on a measurable space $R = > (R,{\mathcal R})$. Then (after extending the sample space {\Omega} if necessary) there exists an $R$-valued random variable $X$ with distribution $\mu$.