Frequentist probability: Can we prove mathematically what we are setting probabilities equal to, or are they just assumptions/definitions?

Question

Just as an example, let's say I am modeling the rolling of a die.

We can use the frequentist definition of probability to define a probability of an event, say rolling a 6, as the $\lim_{n\to\infty}$$\frac{n_6}{n}$, where n is the total number of rolls and $n_6$ is the number of times we obtain a 6.

I usually then see a logical conclusion that, ok this number will obviously be 1/6, in other words, if we roll a die (infinitely) many times, we would get 6 1/6 of the time. So we define P(6) = 1/6.

I am not questioning this logic -- this obviously makes common sense to me. However what I am wondering is -- we theoretically do not know this to be true? i.e., we cannot roll a die infinitely many times. Then would it not be possible that the true limit becomes something else? Basically, are we just logically deducing this to be true (or colloquially, 'as good as true' for our purposes) and defining this to be the probability, or is there actually a mathematical proof deeper in probability theory/statistics that would allow us to draw a conclusion of 1/6 based on the starting assumptions?

Answer 1

The underlying argument for $\frac16$ is essentially based on symmetry of an ideal cube, from which you could argue that you can exploit the various symmetries of faces, edges and vertices to be able to interchange the face-labels without changing the relative outcome probabilities. (You would also need assumptions about uniformity in the physical composition of the die.)

It's a logical argument that is *not actually true of any physical die, since physical dice are subject to imperfect manufacturing processes; they are not exactly cubic (though you can get away with less symmetry than that of a cube, they don't attain that less-demanding symmetry either). The long run frequency will typically show up as some level of unequal probability; the best you could perhaps hope for is you could not distinguish a die from a fair die within the lifetime of the die (many thousands of rolls, perhaps).

Casino dice are manufactured very carefully and are typically so close to fair that you might not be able to readily distinguish their probabilities from uniform in even tens of thousands of rolls (by which time the dice would start to be altered by the repeated throwing), but dice used in some of my boardgames and tabletop roleplaying games are not. I have two wooden dice in my Settlers of Catan game that manage to roll an exceptionally high number of 7's (to the extent that it makes the game pretty much unplayable with those dice, since rolling a 7 has a very particular consequence). I also have a pink and black 20-sided die that turns right angles when you roll it; its bias toward 4 (among other outcomes) is strong enough that you can pick it up quickly, for all that the die is visually very close to symmetric. Presumably there's inconsistent density of material inside the die.

More generally, any model for a process (Bernoulli, Poisson, etc, not just ones based on symmetry arguments) are just that $-$ models. That is, they're approximations of some real process, not exact descriptions.

They abstract away details that you hope are not substantial in their impact, but in practice enough data will reveal imperfections. Take, for example, what is typically treated as one of the most ideal examples we know $-$ treating the number of clicks in a classical Geiger counter for detecting ionizing radiation when pointed at (say) a rock as a homogeneous Poisson process. It isn't quite an accurate model. For one thing, the absolute and relative amounts of different radioactive elements changes over time, and the counter itself doesn't produce two clicks for particles that are very close together in time. So we don't have an actual homogeneous Poisson process for the observed clicks, albeit its typically an excellent approximation.

Since essentially none of these models are actually true, we cannot prove them in the sense of applying to the physical process we observe, even though they do follow from their axioms (/assumptions) and enough data will reveal the failure of assumptions (i.e. a long-enough but very much finite run, rather than the technical sense of "in the long run").

This is not a problem as long as the answers we obtain are close enough for our specific purposes. For a physical die, it won't actually have a probability to roll a $6$ of $\frac16$; but that's not especially important, as long as it's close enough that the model will yield answers that are close enough for our specific purposes.

As George Box elegantly put it "Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful".

Answer 2

You are correct, the assumption of equal probability on a dice roll is a heuristic argument, not to do with science at all. In an experimental setting, we (frequentists) define probability according to the law of large numbers. We therefore concede it's not possible to experimentally obtain such a value, and then we use a pragmatic assumption as to what it is.

A few hundred tosses would usually convince us that 1/6 is a reasonable assumption. Yet we don't actually believe the event probabilities for events {1, 2, 3, 4, 5, 6} to be exactly equal. They are only practically equal. Similarly, no two dice are identical, and if you did the required experimentation - rolls on the order of billions if not trillions to even determine that roll probabilities are not equal - the dice would deform or degrade from use.

Answer 3

It sounds like you are asking about the fundamental difference between probability and statistics.

• In probability, we assume a probability model to be true and we use this to answer questions about specific outcomes.
• In statistics, we acknowledge that there are things we don't know about the probability model, and we use data to try to learn the underlying distribution.

---

To use a die roll as an example, in probability we assume a model for the die, such as

$$p(x) = \begin{cases} \frac{1}{6}, & x\in \{1,2,3,4,5,6\} \\ 0, & \text{otherwise} \end{cases}$$

Of course, this isn't the only valid model. If you have a weighted die, perhaps the model will look like $$p(x) = \begin{cases} \frac{1}{2}, & x=1 \\ \frac{1}{10}, & x\in \{2,3,4,5,6\} \\ 0, & \text{otherwise}. \end{cases}$$ In either case, we can answer questions like the following:

• What is the probability that a roll of the die is at most 2?
• What is the probability that the sum of 3 independent rolls is at least 3?
• etc.

---

In statistics, we start by parameterizing a model (note: there are other ways to do statistics, but this is a common and straightforward approach) for the result of the die $$p(x) = \begin{cases} p_x, & x\in \{1,2,3,4,5,6\} \\ 0, & \text{otherwise}, \end{cases}$$ subject to $\sum_{x=1}^6 p_x = 1$. The next step is to collect data, say $x_1, x_2, \ldots, x_n$ by rolling the die $n$ times and recording the result. This let's us answer questions about the "true" probability model itself.

• Is the "fair die" model consistent with the data?
• What value(s) for $p_1$ is (are) most likely?
• Given our uncertainty about the model, what can we say about the probability questions discussed above?

Answer 4

The fact that there exists a fixed probability (equivalent to the limiting relative frequency of the outcome) for each outcome on the die follows from an assumption of exchangeability using a famous mathematical result called the "representation theorem" (see related answers here, here and here). Essentially, this says that if the outcomes on the die rolls are exchangeable (i.e., the order of a result doesn't affect the overall probability) then those outcomes are independent conditional on some underlying fixed probability for each outcome (which is then equal to the limiting relative frequency). If exchangeability does not hold then there is not a fixed probability for an outcome (see related answer here looking at a coin-flip instead of a die roll).

Absent some strong theoretical reason or empirical analysis, we cannot know that the die roll is "fair" and so the assertion that the probability of a six is $1/6$ is just an assumption. In fact, there are some cases of statistical analysis which allows for some possibility of bias in the allegedly "fair" mechanism. This was examined in a series of papers on the "gambler's fallacy" and broader methods of binomial prediction under the Bayesian paradigm (see O'Neill and Puza 2005; O'Neill 2012; O'Neill 2015). Of course, if we are satisfied that the outcomes of the tosses are exchangeable then we can toss the coin a large number of times to estimate the probability of getting a six, but the idea that this probability is exactly $1/6$ is highly specific and is typically just an assumption.