Do all observations arise from probability distributions?

Question

Below is the quote from Karl Pearson in the book: “The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century” by David Salsburg:

Over a hundred years ago, Karl Pearson proposed that all observations arise from probability distributions and that the purpose of science is to estimate the parameters of those distributions. Before that, the world of science believed that the universe followed laws, like Newton’s laws of motion, and that any apparent variations in what was observed were due to errors. Gradually, Pearson’s view has become the predominant one.

My question is on the use of the word observation. Does the above quote imply that any data we collect or observe in nature/physics/experiments arise from probability distribution? How about deterministic process, which surely in not probabilistic? Any expansion of the above quote for lay person would be very helpful.

Answer 1

Statistics is concerned with phenomena that can be considered random. Even if you are studying a deterministic process, the measurement noise can make the observations random. We can simplify many problems by using simple models that considered all the unobserved factors as “random noise”. For example, the linear regression model

$$ \mathsf{height}_i = \alpha + \beta \,\mathsf{age}_i + \varepsilon_i $$

does say that we model height as a function of age and consider whatever else could influence it as “random noise”. It doesn't say that we consider it as completely “random” meaning “chaotic”, “unpredictable”, etc. For another example, if you toss a coin, the outcome would be deterministic and depend only on the rules of physics, but it is influenced by many factors that contribute to its chaotic nature so we can as well consider it as a random process.

If you have a deterministic process and noiseless measurements of all the relevant data, you wouldn't need statistics for it. You would need other mathematics, for example, calculus, but not statistics. If you need to consider the noise and need to assume randomness, you do so. Nothing “arises” from probability distributions, they are only mathematical tools we use to model real-world phenomena.

Answer 2

Yes, would be the shortest answer. You referred to physics. Physics always disclose measurement errors or precision one way or another. The errors were always a part of the practice of this science. What guys like Pearson did is to treat the errors as random variables. It’s nowadays common practice to follow this approach. Hence, you could say even measurements of deterministic processes are in fact sampling from distributions.

Take a look at the gravitational constant: G here, notice how with the value its uncertainty is given too. Note, this is not an inherently random quantity, it is a constant! Also read the definition of uncertainty in NIST handbook, it is described in terms of probability distributions.

Here a snapshot from a recent physics paper: notice $\pm 0.03$ - the convention to report measurement uncertainty. Physicist sometimes omit it, and when they do it means that all reported digits are significant. For instance, if you see a value “127.010” it means that the uncertainty is around 0.0005, I.e. the last 0 cannot be skipped, because the authors are convinced that it is in fact zero. This is quite different from how quantities are reported in non scientific contexts, where uncertainty goes undisclosed usually

Answer 3

A distribution can be thought of as a data generating function.

When we do inferential statistics, we collect a sample of observations, and then we try to use that sample to figure out the unknown distribution that generated that data.

The reason we want to know the distribution is because we might want to use a model to predict future observations. If we can figure out a good approximation to the true distribution then we can be sure that the future predictions will approximately follow the distribution and we will have a good idea of how accurate our predictions will be.

The distribution doesn't have to be probabilistic. It will still generate data that you can observe, even if the distribution is completely deterministic.

Answer 4

Probability is not fundamentally about the nature of the world (which may, or may not, be deterministic) but about what you know about it.

Consider this example.

• You are sitting with your friends, Alice and Bob.

• I have a standard deck of cards, and shuffle them well.

• What is the probability that the top card is the Ace of spades? Clearly $\frac{1}{52}$.

• I show the top card to Alice, but not to you or Bob.

• If I ask Alice what is the probability that the top card is the Ace of Spades she will surely answer either $1$ or $0$, but not $\frac{1}{52}$.

• But If I ask Bob, he will still have to say $\frac{1}{52}$.

The point is to demonstrate that probability is not fundamentally about reality, but about your knowledge of reality. The cards have not changed their order.

Consider tossing a fair coin. What is the probability that it will land heads? $\frac{1}{2}$?

But in fact tossing a coin is a deterministic process, at least by the standards of modern physics. Scientists have built machines which can predict a coin toss from video of the first milliseconds of motion. Some magicians have trained themselves to toss coins so accurately that they can get either heads or tails at will. I suppose they do this by finely calibrating the force with which the coin is flipped, and the moment at which they catch it, so that they know exactly how many times it has turned over.

But if I, not having trained to do this, were to toss a coin, I couldn't predict the outcome. When I flip a coin it might sometimes go twice or three times as high, or have twice or three times as much angular momentum, as at other times. At best I might say it had turned over between 3 and 15 times. So it is clear that for me, even if I took notice of which way up it was to begin with, my probability will still be close to $\frac{1}{2}$.

Again, the point is not that the process is not deterministic - clearly some scientists and magicians can do it - but that I don't know the parameters of the deterministic function to a high degree of accuracy. My initial ignorance, or imprecision in knowledge, integrated over time, expands out to cover the entire space of possible outcomes, such that I have no idea which way the coin will end up.

Back to your question

My question is on the use of the word observation. Does the above quote imply that any data we collect or observe in nature/physics/experiments arise from probability distribution? How about deterministic process, which surely in not probabilistic?

Go back to the coin toss. The coin toss is in modern terms an entirely deterministic process - if we start with the initial conditions and integrate over time we will get the answer. What makes it "random" is we do not know the initial conditions with enough accuracy to predict if it will be heads or tails. We can take our estimates of the initial conditions and their error bars, and run millions of deterministic Monte-Carlo simulations using slightly different initial conditions, and in each individual simulation there will be an answer, but the answers will be different, and the ratio of heads to tails will be about $\frac{1}{2}$.

So another way of thinking about it is to say that, supposing for a moment that the universe is deterministic, then the "probability distribution" is the weighted distribution of the time integrals of all possible pasts. That is, every possible past - those which we don't know to be false - integrated deterministically through time to the present.

(Kolmogorov will be turning in his grave no doubt.)

So in this view, an observation is a ground truth, and we can integrate backwards through time to eliminate possible pasts which would not give rise to that observation.

• If you have just been dealt the Ace of Hearts, that means I wasn't dealt it earlier.

In summary,

• I would not say that an observation arises from a probability distribution.
• There is a reality out there which gives rise to observations.
• Observations give us information about that reality (specifically about the past) and we can combine observations to create a model of the past which allows us to make predictions about the future.
• That model of the past is the probability distribution referred to.

Answer 5

Below is the quote from Karl Pearson

You are not placing an exact quote of Karl Pearson, so it is difficult to respond directly to the thinking of Pearson. His thinking was positivist/idealistic and his ideas where like physical laws being relative to the observations by humans.

• It is not observations that arise from probability distributions as if the underlying laws that govern them need to be probabilistic.

  Instead it is more like the other way around. The laws of nature stem from observations (and these observations happen to be of a probabilistic nature due to variations that occur in experiments).

• It is not that observations arise from probability distributions, but they follow probability distributions. Finding out whether the underlying 'reality' is deterministic or not, that is not the goal of a positivist science, because it can only use the observations. These observations happen to have a random behaviour but whether their nature is random or deterministic is outside the grasp of science. So science should focus on describing the parameters of these distributions and not about the meaning or cause behind it (which would be metaphysics).

The difference with Newtonian notions or contemporary biologist in the time of Pearson is that science should be about data (observations) and not about unverifiable theories and notions about reality. Science is about measurements and that makes science a practice that deals with statistics (the field that is about description and analysis of data).

This question is actually more about philosophy of science than about statistics.