21
$\begingroup$

I am trying to better understand the Difference in "Probability Measure" and "Probability Distribution"

I came across this post, which says

The difference between the terms "probability measure" and "probability distribution" is in some ways more of a difference in connotation of the terms rather than a difference between the things that the terms refer to. It's more about the way the terms are used. "*

The answer over here suggests that these two concepts might be the same thing! So, my question is: can we consider the "Normal Probability Distribution Function" as a "Probability Measure"?

$\endgroup$

7 Answers 7

22
$\begingroup$

First off, I am not used to the term "Probability Distribution Function". In case you are referring to a "PDF" when you are using "Probability Distribution Function", then I would like to point out that PDF is rather the abbreviation for "Probability Density Function". Below, I will presume you meant probability density function.

Second, the word "distribution" is used very differently by different people, but I will refer to the definition of this notion in the scientific community.

In a nutshell: The distribution of a random variable $X$ is a measure on $\mathbb{R}$, while the PDF of $X$ is a function on $\mathbb{R}$ and the PDF doesn't even always exist. So they are very different.

And now we get to the mathematical details: First, let's define the term random variable because that is what all those terms usually refer to (unless you get a step further and want to talk about random vectors or random elements, but I will restrict this here to random variables). I.e., you talk about the distribution of a random variable.

Given a probability space $(\Omega, \cal{F}, p)$ ($\Omega$ is just a set, $\cal{F}$ is a sigma algebra on $\Omega$, and $p$ is a measure on $(\Omega, \cal{F})$), a random variable $X$ is a measureable map $X: \Omega \to \mathbb{R}$.

Then we can define: The distribution $p_X$ of the random variable $X$ is the measure $p_X = p \circ X^{-1}$ on $\mathbb{R}.$

I.e. you push the measure $p$ from $\Omega$ forward to $\mathbb{R}$ via the measurable function $X$.

Next we define the probability density function (PDF) of a random variable $X$: The PDF $f_X$ of a random variable $X$, if it exists, is the Radon-Nikodym derivative of its distribution w.r.t. the Lebesgue measure $\lambda$, i.e. $f_X = \frac{d\,p_X}{d\,\lambda}$.

So the distribution $p_X$ and the PDF $f_X$ of a random variable are very different entities (to a stickler, at least). But very often, the PDF $f_X$ exists, and then it contains all of the relevant information about the distribution $p_X$ and can thus be used as a handy substitute for the unwieldy $p_X$.

$\endgroup$
5
  • 1
    $\begingroup$ The pdf if it exists induces the CDF which in turn induces the distribution p_X uniquely. This is so in all the cases. $\endgroup$
    – user266286
    Commented Mar 12, 2022 at 4:37
  • 1
    $\begingroup$ Sure. I did not mean to claim otherwise. I edited the last sentence slightly to remove any chance of misinterpretation. $\endgroup$
    – frank
    Commented Mar 5, 2023 at 4:24
  • 2
    $\begingroup$ <This comment is with a gentle tone> While this response "answers" the question, I doubt that it adds any illumination for a person inquiring about this topic and first learning statistics. In order to understand what is written here, one must already know the answer to the question. $\endgroup$ Commented Jun 20, 2023 at 23:59
  • $\begingroup$ @frank - I think you missed talking about "Probability Measure", what is it and how is it different from a probability density function? $\endgroup$
    – x.projekt
    Commented Jul 3, 2023 at 9:48
  • $\begingroup$ Page 364 in Levin&Peres, Markov Chains and Mixing Times, second edition (pages.uoregon.edu/dlevin/MARKOV/markovmixing.pdf). If $\Omega$ is a countable set, a probability distribution (or sometimes simply a probability) on $\Omega$ is a function $p : \Omega \rightarrow [0, 1]$ such that \begin{equation} \sum_{\xi \in \Omega} p(\xi) = 1. \end{equation} We will abuse notation and write, for any subset, $A \subset \Omega$, \begin{equation} p(A)=\sum_{\xi \in A} p(\xi). \end{equation} The set function $A \mapsto p(A)$ is a probability measure. $\endgroup$
    – Ommo
    Commented Sep 17, 2023 at 17:33
12
$\begingroup$

To navigate the complexities of probability theory, it's helpful to start with a foundational vocabulary, accessible to those not deeply versed in professional mathematics.

In Functional Analysis:

  • Distributions: In mathematics, particularly in functional analysis, distributions are advanced objects that generalize functions. While the intricacies of distributions go beyond a basic understanding, it's important to recognize their role in the broader mathematical landscape.

  • Density: This concept emerges from distributions under certain conditions. In ideal scenarios, densities can be thought of as the derivatives of distributions, offering a more granular look at how distributions behave across different points.

In Probability Theory:

  • Cumulative Distribution Function (CDF): This function represents a normalized version of a distribution. It ensures that the total probability across the entire domain (e.g., across all real numbers) sums up to 1, highlighting the probabilistic nature of the distribution.

  • Probability Density Function (PDF): Similar to the CDF, the PDF is focused on the cumulative aspects of distributions but from the lens of density, providing a way to understand how probability is distributed across different values.

In Measure Theory:

  • Random Variable: At its core, a random variable is a measurable function, though this might initially seem abstract. For simplicity, consider it as a specific type of function that aligns with certain mathematical criteria for measurability.

  • Measure: Conceptually, a measure is a way to assign a "weight" or numerical value to sets, essentially quantifying them in some meaningful way, adhering to a set of mathematical rules.

  • Probability Measure: This is a special case of a measure where the total measure (or weight) of the entire space is 1, aligning with the foundational principle of probability.

Key Theorems and Insights:

  • Association of Random Variables and CDFs: The relationship between a random variable and its Cumulative Distribution Function (CDF) is inseparable. Defining one implicitly defines the other, reflecting a deep interconnectedness.

  • CDFs and Probability Measures: For every CDF, a unique probability measure exists. This theorem bridges the gap between abstract mathematical concepts and their practical application in probability theory, underscoring a fundamental link.

  • PDFs and CDFs: Under conditions where the Probability Density Function (PDF) directly relates to the CDF (through differentiation and integration), the PDF also uniquely characterizes the random variable. This relationship might not hold in all cases, especially when direct equivalence between PDF and CDF is absent.

Conclusion:

In light of these concepts and theorems, it's clear that probability distributions and measures are closely related. Understanding one concept allows for the construction or comprehension of the other, facilitating a more streamlined approach to dealing with probabilistic models. Good luck on this path of knowledge!

$\endgroup$
3
  • 1
    $\begingroup$ @RuiBarradas - a minor point, but the integral of the CDF is bounded if the variate has an upper bound, e.g., a Uniform$(0,1)$ variate. $\endgroup$
    – jbowman
    Commented Feb 12, 2022 at 19:57
  • $\begingroup$ @jbowman Yes, thanks. And that's not a minor point. $\endgroup$ Commented Feb 12, 2022 at 22:09
  • $\begingroup$ @RuiBarradas - I was thinking it was minor because your statement was based on an oversight, not on a misunderstanding. $\endgroup$
    – jbowman
    Commented Feb 13, 2022 at 0:04
11
$\begingroup$

The concept of a "probability distribution" is an umbrella term that refers to a particular type of object that can be represented uniquely in multiple ways. One way to represent a probability distribution is through its probability measure, another is through its characteristic function, another is through its cumulative distribution function, and another is through its probability density function (including specification of a dominating measure for the density). All of the latter are specific mathematical objects that describe a probability distribution in a different way. The term "probability distribution" does not refer to a specific mathematical object; it can be thought of as an umbrella term that refers holistically to the "thing" that each of these objects is describing.

$\endgroup$
0
5
$\begingroup$

You are right, when we are just starting out our stat studies, these terms can be super confusing, especially to people that are very detail-driven or very particular about terminologies. You do need a very good understanding here before you can master the applications of them. And using R functions such as pnorm(), dnorm() forces you to understand these.

And I’d say the post you quoted put it pretty well - especially where it pointed out that “probability functions” - “probability density function” and “probability mass function” are precisely defined, and the other terms can be understood as their names suggest - that the “probability measure” is a measure for the probabilities - is it cumulative or not? It doesn’t say. It’s a measure, so it can refer to both. A probability distribution can be similar. But don't confuse it with distribution function. Distribution function in most cases refers to "cumulative distribution function", which is also clearly defined as the probability taking on a value equal to or less than a specified value.

Maybe an example can help

enter image description here

This is a simple normal distribution chart. And here,

  • The probability function - which is a probability density function here since it’s continuous variable - describes the red curve. It’s a line, that’s represented by this function $$\large{f(x)=\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}(\frac{x-\mu}{\sigma})^2}}$$ where $\mu$ is the mean and $\sigma$ the standard deviation.

  • Calculating integrals under this “red line” gets us different “probability” values - for example, the orange area is an area that represents 1 standard deviation away from the mean with 34% area of the total - which is 34% probability; whereas the blue area represents 17% probability. I’d say the entire chart depicting the probabilities with a normal distribution is a probability measure and a probability distribution. But only and precisely the red line is a probability function, or more specifically, a probability density function.

Don't stress too much about what exactly is the definition for "probability measure"/"probability distribution". I believe as you see more examples, you will feel comfortable using these terms.

$\endgroup$
5
  • 3
    $\begingroup$ Most authors in math and stats make a clear distinction between the distribution function, also known as a CDF, and the probability density function, or PDF. Your use of "probability distribution" and your equation of measures with distribution functions ("it can refer to both") looks unusual and might be confusing to those familiar with the standard terminology. $\endgroup$
    – whuber
    Commented Feb 11, 2022 at 22:23
  • $\begingroup$ Of course, distribution function and probability density function are not the same at all. As shared in my explanation, the probability density function is clearly and precisely defined. $\endgroup$
    – Peiran Yu
    Commented Feb 12, 2022 at 3:15
  • $\begingroup$ CDF is also a lot of times represented by the term "distribution function". I haven't found a good definition for probability measure yet. Some say it's a function. // All in all, I think we all agree probability functions are well defined and clear. The other terms aren't always so much. Would love to know a good source of standard definitions if you know of them! $\endgroup$
    – Peiran Yu
    Commented Feb 12, 2022 at 3:19
  • $\begingroup$ Walter Rudin, Real and Complex Analysis, is the classic account of measures. Various approaches are possible from the perspectives of measure theory or functional analysis (where measures appear as elements of the dual space to $L^\infty(\mathbb R),$ effectively defining probability measure in terms of expectation). $\endgroup$
    – whuber
    Commented Feb 12, 2022 at 15:47
  • $\begingroup$ @PeiranYu: "Some say it is a function". Any measure, not just probability measure, is a function, so that is technically correct. $\endgroup$
    – user266286
    Commented Mar 12, 2022 at 4:41
2
$\begingroup$

A probability measure $P$ is a function which associates each event with its probability.

If the events are real Borel sets (nice sets of real numbers, such as intervals, finite sets etc) then in lieu of describing the probability measure (a task which is directly unwieldy) we can instead use a special kind of real valued function $F$ defined on the reals. This function is called as distribution function (or simply distribution) and it is defined so as to make $F(x)=P(-\infty,x]$. A mathematical theorem then guarantees the unique extension to the probabilities of other Borel sets.

From a probability theory perspective, events are not Borel sets most of the time. However we may associate outcomes with real numbers (usually by some measurement) - this in turn induces an association of 'the practically useful events' to Borel sets; after which we proceed to use a distribution to describe the probabilities of these Borel sets.

$\endgroup$
3
  • $\begingroup$ This was very good until the last paragraph, where it seems to fall apart into a stream of contradictory and incorrect statements. Maybe there are some typographical errors there? $\endgroup$
    – whuber
    Commented Mar 12, 2022 at 18:32
  • $\begingroup$ I am referreing to the fact that an RV associates each outcome to a real number and useful events $(X\in B)$ are associated thereby to Borel sets. The probabilities are now defined for Borel sets in practice. $\endgroup$
    – user266286
    Commented Mar 13, 2022 at 0:55
  • $\begingroup$ See the updated post above. $\endgroup$
    – user266286
    Commented Mar 13, 2022 at 2:27
1
$\begingroup$

Probability measure can be seen as a generalization of probability density function (PDF). Probability distribution function is not really a term, but when I saw it used it was always meant to refer to the cumulative distribution function (CDF). I wouldn't use this as a term, since it's confusing.

In Excel NORM.S.DIST() function has an argument that switches it between cumulative and density functions. To make things more confusing Microsoft called the density function as "probability mass function", which is obviously wrong because PMF is for discrete distributions.

It's ok to use probability distribution term since it references everything we know about the distribution, including PDF and CDF and moments etc.

$\endgroup$
1
$\begingroup$

This is my 2 cents, though I'm not an expert:

"Probability Measure" is used in the context of a more precise, math theoretical, context. Kolmogorov in the year 1933 laid down some mathematical constructs to help better understand and handle probabilities from a mathematically rigid point of view. In a nutshell - he defined a "Probability Space" which consists of a set of events, a (σ)-algebra/field on that set (≈ all the different ways you can subset that original set), and a measure which maps these subsets to a number that measures them. This became the standard way of understanding probability. This framework is important because once you start thinking about probability the way mathematicians do, you encounter all kind of edge cases and problems - which the framework can help you define or avoid.

So, I would say that people who use "Probability Measure" are either involved with deep probability issues, or are simply more math oriented by their education.

Note that a "Probability Space" precedes a "Random Variable" (also known as a "Measurable Function") - which is defined to be a function from the original space to measurable space, often real-valued. I'm not sure, but I think the main point here, is that this allows us to use more "number-oriented" math, than "space-oriented" math. We map the "space" into numbers, and now we can work more easily with it. (There's nothing to prevent us to start with a "number space", e.g., $\mathbb R$ and define the identity mapping as the Random Variable; But a lot of events are not intrinsically numbers - think of Heads or Tails, and the mapping of them into numbers 0 or 1).

Once we are in the realm of numbers (real line R), we can define Probability Functions to help us characterize the behavior of these fantastic probability beasts. The main function is called the "Cumulative Distribution Function" (CDF) - it exists for all valid probability spaces and for all valid random variables, and it completely defines the behavior of the beast (unlike, say, the mean of a random variable, or the variance: you can have different probability beasts with the same mean or the same variance, and even both). It keeps tracks on how much the probability measure is distributed across the real line.

If the random variable mapping is continuous, you will also have a Probability Density Function (PDF), if it's discrete you will have a Probability Mass Function (PMF). If it's mixed, it's complicated.

I think "Probability Distribution" might mean either of these things, but I think most often it will be used in less mathematically precise as it's sort of an umbrella term - it can refer to the distribution of measure on the original space, or the distribution of measure on the real line, characterized by the CDF or PDF/PMF.

Usually, if there's no need to go deep into the math, people will stay on the level of "probability function" or "probability distribution". Though some will venture to the realms of "probability measure" without real justification except the need to be absolutely mathematically precise.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.