20
$\begingroup$

I am trying to better understand the Difference in "Probability Measure" and "Probability Distribution"

I came across this post, which says

The difference between the terms "probability measure" and "probability distribution" is in some ways more of a difference in connotation of the terms rather than a difference between the things that the terms refer to. It's more about the way the terms are used. "*

The answer over here suggests that these two concepts might be the same thing! So, my question is: can we consider the "Normal Probability Distribution Function" as a "Probability Measure"?

Improve this question
$\endgroup$

7 Answers 7

Reset to default
20
$\begingroup$

First off, I am not used to the term "Probability Distribution Function". In case you are referring to a "PDF" when you are using "Probability Distribution Function", then I would like to point out that PDF is rather the abbreviation for "Probability Density Function". Below, I will presume you meant probability density function.

Second, the word "distribution" is used very differently by different people, but I will refer to the definition of this notion in the scientific community.

In a nutshell: The distribution of a random variable $X$ is a measure on $\mathbb{R}$, while the PDF of $X$ is a function on $\mathbb{R}$ and the PDF doesn't even always exist. So they are very different.

And now we get to the mathematical details: First, let's define the term random variable because that is what all those terms usually refer to (unless you get a step further and want to talk about random vectors or random elements, but I will restrict this here to random variables). I.e., you talk about the distribution of a random variable.

Given a probability space $(\Omega, \cal{F}, p)$ ($\Omega$ is just a set, $\cal{F}$ is a sigma algebra on $\Omega$, and $p$ is a measure on $(\Omega, \cal{F})$), a random variable $X$ is a measureable map $X: \Omega \to \mathbb{R}$.

Then we can define: The distribution $p_X$ of the random variable $X$ is the measure $p_X = p \circ X^{-1}$ on $\mathbb{R}.$

I.e. you push the measure $p$ from $\Omega$ forward to $\mathbb{R}$ via the measurable function $X$.

Next we define the probability density function (PDF) of a random variable $X$: The PDF $f_X$ of a random variable $X$, if it exists, is the Radon-Nikodym derivative of its distribution w.r.t. the Lebesgue measure $\lambda$, i.e. $f_X = \frac{d\,p_X}{d\,\lambda}$.

So the distribution $p_X$ and the PDF $f_X$ of a random variable are very different entities (to a stickler, at least). But very often, the PDF $f_X$ exists, and then it contains all of the relevant information about the distribution $p_X$ and can thus be used as a handy substitute for the unwieldy $p_X$.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ The pdf if it exists induces the CDF which in turn induces the distribution p_X uniquely. This is so in all the cases. $\endgroup$
    – Shahab
    Mar 12, 2022 at 4:37
  • 1
    $\begingroup$ Sure. I did not mean to claim otherwise. I edited the last sentence slightly to remove any chance of misinterpretation. $\endgroup$
    – frank
    Mar 5 at 4:24
  • 1
    $\begingroup$ <This comment is with a gentle tone> While this response "answers" the question, I doubt that it adds any illumination for a person inquiring about this topic and first learning statistics. In order to understand what is written here, one must already know the answer to the question. $\endgroup$
    – Ricardo Saporta
    Jun 20 at 23:59
  • $\begingroup$ @frank - I think you missed talking about "Probability Measure", what is it and how is it different from a probability density function? $\endgroup$
    – x.projekt
    Jul 3 at 9:48
  • $\begingroup$ Page 364 in Levin&Peres, Markov Chains and Mixing Times, second edition (pages.uoregon.edu/dlevin/MARKOV/markovmixing.pdf). If $\Omega$ is a countable set, a probability distribution (or sometimes simply a probability) on $\Omega$ is a function $p : \Omega \rightarrow [0, 1]$ such that \begin{equation} \sum_{\xi \in \Omega} p(\xi) = 1. \end{equation} We will abuse notation and write, for any subset, $A \subset \Omega$, \begin{equation} p(A)=\sum_{\xi \in A} p(\xi). \end{equation} The set function $A \mapsto p(A)$ is a probability measure. $\endgroup$
    – Ommo
    Sep 17 at 17:33
11
$\begingroup$

The concept of a "probability distribution" is an umbrella term that refers to a particular type of object that can be represented uniquely in multiple ways. One way to represent a probability distribution is through its probability measure, another is through its characteristic function, another is through its cumulative distribution function, and another is through its probability density function (including specification of a dominating measure for the density). All of the latter are specific mathematical objects that describe a probability distribution in a different way. The term "probability distribution" does not refer to a specific mathematical object; it can be thought of as an umbrella term that refers holistically to the "thing" that each of these objects is describing.

Improve this answer
$\endgroup$
0
9
$\begingroup$

You mention the post from mathstack. The answers there are great and they should be self-sufficient. I recommend anyone to go read it if they are interested in the maths details.

If you are asking here the "same" question, this is probably because you are not familiar with the terminology and the mathematics of probability theory.

What I do not like about the other answers is that they are focusing on the densities instead of what is really important: distributions and measures.

For this reason, here is some vocabulary, aimed at non professional mathematicians. It is a very quick and dirty presentation:

Functional analysis:

  • Distributions is a mathematical object that you develop in functional analysis and you do not need to know the details here.

  • Density is the concept that arises FROM distribution in nice scenarios. When everything is nice, densities are the derivatives of distributions.

Probability Theory:

  • Cumulative Distribution Function, it is a "normalised" distribution, and for this reason its value over the whole domain is equal to $1$ ( over $ \mathbb R $ if you wish, $ \lim_{x \to \infty } F_X( x ) = 1$),

  • Probability Density Function, same as above, but for Cumulative Distributions.

Measure Theory:

  • Random Variable: it is essentially a measurable function. If you do not understand this, skip it, and come back later to this topic. For now consider it as a function. All measurable functions are functions, but all functions are not measurable. However, in most situations, what you could think about is measurable.

  • Measure: a measure is a function from sets to reals. In other words, it attributes a weight to sets of elements. Measures have to respect some properties that I do not detail here.

  • Probability Measure: a probability measure is a normalized measure such that the measure of the whole space equals to $1$.

Now that we know the vocabulary, the important theorems you should be aware are:

  1. A random variable is ALWAYS associated to a CDF (cumulative distribution function). It is not possible to have one without the other. If I explicitly define $X$ you can find the CDF of $X$ coined $F_X$ and inversely.

  2. For every CDF, there exists a unique associated probability measure.

What does it mean? it means that if you start with a random variable, you have a CDF, which corresponds to a measure! And inversely!

  1. A bonus, when the PDF (probability density function) is equivalent to the CDF (in the sense that the derivative of the CDF is equal to the PDF and the anti-derivative of the PDF gives the CDF), then what I said about the CDF (that it uniquely characterises the random variable) is also true for the PDF! Be careful to cases when the PDF and CDF are not equivalent.

The conclusion is that yes, distributions and measures are equivalent. If you have one, you can construct the other. This is great because it simplifies many cases where you can just work out the expression that is easier to work with.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ @RuiBarradas - a minor point, but the integral of the CDF is bounded if the variate has an upper bound, e.g., a Uniform$(0,1)$ variate. $\endgroup$
    – jbowman
    Feb 12, 2022 at 19:57
  • $\begingroup$ @jbowman Yes, thanks. And that's not a minor point. $\endgroup$
    – Rui Barradas
    Feb 12, 2022 at 22:09
  • $\begingroup$ @RuiBarradas - I was thinking it was minor because your statement was based on an oversight, not on a misunderstanding. $\endgroup$
    – jbowman
    Feb 13, 2022 at 0:04
5
$\begingroup$

You are right, when we are just starting out our stat studies, these terms can be super confusing, especially to people that are very detail-driven or very particular about terminologies. You do need a very good understanding here before you can master the applications of them. And using R functions such as pnorm(), dnorm() forces you to understand these.

And I’d say the post you quoted put it pretty well - especially where it pointed out that “probability functions” - “probability density function” and “probability mass function” are precisely defined, and the other terms can be understood as their names suggest - that the “probability measure” is a measure for the probabilities - is it cumulative or not? It doesn’t say. It’s a measure, so it can refer to both. A probability distribution can be similar. But don't confuse it with distribution function. Distribution function in most cases refers to "cumulative distribution function", which is also clearly defined as the probability taking on a value equal to or less than a specified value.

Maybe an example can help

enter image description here

This is a simple normal distribution chart. And here,

  • The probability function - which is a probability density function here since it’s continuous variable - describes the red curve. It’s a line, that’s represented by this function $$\large{f(x)=\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}(\frac{x-\mu}{\sigma})^2}}$$ where $\mu$ is the mean and $\sigma$ the standard deviation.

  • Calculating integrals under this “red line” gets us different “probability” values - for example, the orange area is an area that represents 1 standard deviation away from the mean with 34% area of the total - which is 34% probability; whereas the blue area represents 17% probability. I’d say the entire chart depicting the probabilities with a normal distribution is a probability measure and a probability distribution. But only and precisely the red line is a probability function, or more specifically, a probability density function.

Don't stress too much about what exactly is the definition for "probability measure"/"probability distribution". I believe as you see more examples, you will feel comfortable using these terms.

Improve this answer
$\endgroup$
5
  • 3
    $\begingroup$ Most authors in math and stats make a clear distinction between the distribution function, also known as a CDF, and the probability density function, or PDF. Your use of "probability distribution" and your equation of measures with distribution functions ("it can refer to both") looks unusual and might be confusing to those familiar with the standard terminology. $\endgroup$
    – whuber ♦
    Feb 11, 2022 at 22:23
  • $\begingroup$ Of course, distribution function and probability density function are not the same at all. As shared in my explanation, the probability density function is clearly and precisely defined. $\endgroup$
    – Peiran Yu
    Feb 12, 2022 at 3:15
  • $\begingroup$ CDF is also a lot of times represented by the term "distribution function". I haven't found a good definition for probability measure yet. Some say it's a function. // All in all, I think we all agree probability functions are well defined and clear. The other terms aren't always so much. Would love to know a good source of standard definitions if you know of them! $\endgroup$
    – Peiran Yu
    Feb 12, 2022 at 3:19
  • $\begingroup$ Walter Rudin, Real and Complex Analysis, is the classic account of measures. Various approaches are possible from the perspectives of measure theory or functional analysis (where measures appear as elements of the dual space to $L^\infty(\mathbb R),$ effectively defining probability measure in terms of expectation). $\endgroup$
    – whuber ♦
    Feb 12, 2022 at 15:47
  • $\begingroup$ @PeiranYu: "Some say it is a function". Any measure, not just probability measure, is a function, so that is technically correct. $\endgroup$
    – Shahab
    Mar 12, 2022 at 4:41
2
$\begingroup$

A probability measure $P$ is a function which associates each event with its probability.

If the events are real Borel sets (nice sets of real numbers, such as intervals, finite sets etc) then in lieu of describing the probability measure (a task which is directly unwieldy) we can instead use a special kind of real valued function $F$ defined on the reals. This function is called as distribution function (or simply distribution) and it is defined so as to make $F(x)=P(-\infty,x]$. A mathematical theorem then guarantees the unique extension to the probabilities of other Borel sets.

From a probability theory perspective, events are not Borel sets most of the time. However we may associate outcomes with real numbers (usually by some measurement) - this in turn induces an association of 'the practically useful events' to Borel sets; after which we proceed to use a distribution to describe the probabilities of these Borel sets.

Improve this answer
$\endgroup$
3
  • $\begingroup$ This was very good until the last paragraph, where it seems to fall apart into a stream of contradictory and incorrect statements. Maybe there are some typographical errors there? $\endgroup$
    – whuber ♦
    Mar 12, 2022 at 18:32
  • $\begingroup$ I am referreing to the fact that an RV associates each outcome to a real number and useful events $(X\in B)$ are associated thereby to Borel sets. The probabilities are now defined for Borel sets in practice. $\endgroup$
    – Shahab
    Mar 13, 2022 at 0:55
  • $\begingroup$ See the updated post above. $\endgroup$
    – Shahab
    Mar 13, 2022 at 2:27
1
$\begingroup$

Probability measure can be seen as a generalization of probability density function (PDF). Probability distribution function is not really a term, but when I saw it used it was always meant to refer to the cumulative distribution function (CDF). I wouldn't use this as a term, since it's confusing.

In Excel NORM.S.DIST() function has an argument that switches it between cumulative and density functions. To make things more confusing Microsoft called the density function as "probability mass function", which is obviously wrong because PMF is for discrete distributions.

It's ok to use probability distribution term since it references everything we know about the distribution, including PDF and CDF and moments etc.

Improve this answer
$\endgroup$
1
$\begingroup$

This is my 2 cents, though I'm not an expert:

"Probability Measure" is used in the context of a more precise, math theoretical, context. Kolmogorov in the year 1933 laid down some mathematical constructs to help better understand and handle probabilities from a mathematically rigid point of view. In a nutshell - he defined a "Probability Space" which consists of a set of events, a (σ)-algebra/field on that set (≈ all the different ways you can subset that original set), and a measure which maps these subsets to a number that measures them. This became the standard way of understanding probability. This framework is important because once you start thinking about probability the way mathematicians do, you encounter all kind of edge cases and problems - which the framework can help you define or avoid.

So, I would say that people who use "Probability Measure" are either involved with deep probability issues, or are simply more math oriented by their education.

Note that a "Probability Space" precedes a "Random Variable" (also known as a "Measurable Function") - which is defined to be a function from the original space to measurable space, often real-valued. I'm not sure, but I think the main point here, is that this allows us to use more "number-oriented" math, than "space-oriented" math. We map the "space" into numbers, and now we can work more easily with it. (There's nothing to prevent us to start with a "number space", e.g., $\mathbb R$ and define the identity mapping as the Random Variable; But a lot of events are not intrinsically numbers - think of Heads or Tails, and the mapping of them into numbers 0 or 1).

Once we are in the realm of numbers (real line R), we can define Probability Functions to help us characterize the behavior of these fantastic probability beasts. The main function is called the "Cumulative Distribution Function" (CDF) - it exists for all valid probability spaces and for all valid random variables, and it completely defines the behavior of the beast (unlike, say, the mean of a random variable, or the variance: you can have different probability beasts with the same mean or the same variance, and even both). It keeps tracks on how much the probability measure is distributed across the real line.

If the random variable mapping is continuous, you will also have a Probability Density Function (PDF), if it's discrete you will have a Probability Mass Function (PMF). If it's mixed, it's complicated.

I think "Probability Distribution" might mean either of these things, but I think most often it will be used in less mathematically precise as it's sort of an umbrella term - it can refer to the distribution of measure on the original space, or the distribution of measure on the real line, characterized by the CDF or PDF/PMF.

Usually, if there's no need to go deep into the math, people will stay on the level of "probability function" or "probability distribution". Though some will venture to the realms of "probability measure" without real justification except the need to be absolutely mathematically precise.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.