Can we think of a probability in both the classical and subjective sense simultaneously?

Question

I'm a statistics student. I am trying to understand the classical and objective definitions of probability and how they are related to frequentist and Bayesian inference. It's not obvious to me why classical probability is paired with frequentist inference and why Bayesian inference is paired with subjective probability. In some sources I've read statements like the following from this paper by Wellek (sorry I couldn't find a version that wasn't behind a pay wall):

From the frequentist viewpoint, population parameters are unobservable constants about which no meaningful probability statements can be made.

I am trying to understand if this is due to the classical definition of probability as repeated trials or if it is due to the constraints of frequentist inference.

My specific questions are at the end if readers would prefer to skip ahead, but I wanted to share my thoughts in case that helps.

Consider the random variable $X$. Under the classical definition of probability if I wanted to scientifically empirically measure the probability that $P(X=x)$, I would think I just have to repeat the experiment a large number of times and make a tally. By the subjective definition, I think am expected to initially consult my own beliefs or the beliefs of a rational agent. As I collect more data, those beliefs are then rationally modified.

Now it seems to me $H_0|X$ can't be observed so there is no way to compute the value of $P(H_0|X)$ by my classical empirical procedure. In contrast, I can always believe in things I can't directly observe such as $H_0|X$ and so assuming I know some relationship between things I can observe like $X$ and things I can't observe like $H_0$, this allows me to have beliefs about $P(H_0|X)$ which I can rationally modify over time.

It occurs to me that I can also argue that for the frequentist, $H_0$ is a fixed property of the universe anyway so I am perhaps stuck with the notion that $H_0$ is fixed even if I could observe it. But what if we think about the typical experiment of flipping a coin and we modify it to say I have a large supply of quarters and I always use a new one every time I record a flip. So in that case, I suspect there is an underlying parameter $p$ which is coin specific but I can never directly observe it. Therefore, $P(p=0.5|X)$ is meaningful but I can't compute it via direct observation of $p$.

So just to circle back to my high-level questions.

1. Is there any meaningful way to interpret a Bayesian inference procedure as a frequentist?

2. Is there any meaningful way to do Bayesian inference where the probabilities are defined according to the classical definition of probability?

Answer 1

There is an earlier very similar question at: Is probability fundamentally about reference classes (real or imagined)? and I am really trying here to answer both. We must start with what is probability?

The mathematical answer is that whatever fulfills the usual probability axioms is a probability. But that does not really answer our question! because we are at how can we use probability, how can we interpret probability. In my thinking (today, 7 mars 2018, tomorrow ...) all conceptions of probability (frequentist, subjective, personal, ...) have one important thing in common, which gives them a common meaning. That is the concept of calibration or reference classes. For calibration see for instance Visualizing the calibration of predicted probability of a model or search this site for "calibration". For reference classes the the first linked post.

Calibration differs in what kind of reference classes you are calbrating against:

1. For classical frequentist probability we are calibrating against a well-defined set of empirically observable and similar events, like repeats of throws of the same coin or dice. These probabilities are said to be objective because most reasonable persons will agree on them.
2. Bookies need to calibrate against their pool of punters, so that in the long run his wins and losses are balanced. If his pool of punters, as a collective, is badly calibrated in the sense of 1), the bookie must adjust to that. Another way of saying this is that the bookies odds (or probabilities) are really market prices.
3. Subjective probability is formed by the person making the judgement forming equivalence classes of events with uncertain outcome. Events $A, B$ are in the same equivalence class if that person judges them as exchangeable. That means that for this person, in any system of bets involving $A$ but not $B$ he can exchange $A$ for $B$ and still evaluate the bet at the same price. Such events must have the same probability. So the equivalence classes consists of events judged to have the same probability, and the calibration is against the long run frequency of events in the class.

We could say that probability has a Janus face:

(or maybe it is better to say that probability is a many-headed troll!)

With this viewpoint it seems like there is a continuum of interpretations, from very objective to very subjective! more than discrete different interpretations. A lot of practical discussions about probability can be framed within this concepts, an example: What is the probability of a blowout in the North Sea? Just prior to the Bravo blowout at the Ekofisk field this was discussed at the Storting, one member asked what had been done about blowout risk, and mentioned probabilities taken from historical experience from the Mexican gulf. The oil minister Bjartmar Gjerde responded that this reminded him of the woman who had three children, and did not want more, because she had heard that each fourth child born was Chinese ... The day after the blowout the newspaper dagbladet had this title: "A Chinese is born in the North Sea".

So, should experience from the Mexican Gulf be used in forming reference classes for the North Sea, or formulated otherwise, should blowouts in the Gulf of Mexico and in the North Sea be calibrated together or separately? There is no completely objective answer to such questions, making such evaluations partly subjective even if we aim at frequency interpretations.

So "subjectivity" in "subjective probability" means that different reasonable people can come to different evaluations, define reference classes in different ways, not that "everything goes". For subjective probability to be meaningful it is expected that the reference classes (or calibration procedure) is made as explicit as possible, so that discussion and criticism is made possible. Otherwise it will just degenerate to a game of guess-a-number.

So, in this sense (subjective) probability is only meaningful for events which probability can be calibrated. One example where this is not possible (do I believe) is $\DeclareMathOperator{\P}{\mathbb{P}}\P(\text{Does God exist})$. Calibration is impossible, so such a probability cannot be made meaningful. In this sense, we can talk about (subjective) probabilty of event $E$ only if the occurrence or not of $E$ will eventually be known, or at least that that is possible, at least in principle.

So to your first question:

From the frequentist viewpoint, population parameters are unobservable constants about which no meaningful probability statements can be made.

Well, the frequentist chooses to not make any probability statements about the unknown (but fixed) parameter value, but is he forced to do that choice? Now, he is not. As an example, let the parameter $\mu$ represent the average height of 20 year old Norwegian males. We don't know the exact value, so it makes sense to set up bets on events (propositions) involving $\mu$, say $E=\{ \mu <= 1.73\text{m}\}$. But above we said that "... we can talk about (subjective) probability of event $E$ only if the occurrence or not of $E$ will eventually be known, ...". Is that the case of $E$? Not as formulated, but bayesians like deFinetti has thought deeply about this problem, and come forth with the following resolution. Let $X=(X_1, X_2, \dotsc, X_n, \dotsc)$ be observation of height on a random sample of 20 yo Norwegian males. Before taking the sample, we will have some expectations about the possible values. Being a random sample, the distribution of $X$ will be exchangeable. deFinetti proved his representation theorem, an (infinitely) exchangeable $X$ can be represented as conditionally independent given some underlying, latent variable, in the example $\mu$ (and maybe some other parameters as $\sigma$), and with a prior distribution on this latent variables/parameters. In this way, we can construct a (prior) probability distribution on $\mu$, and still think about it as an unknown constant! That can look strange, but can be resolved by thinking of this distribution as epistemic, not aleatori. In this way the bet on $\mu$ can be translated to a bet on $X$, which we can resolve.

(not really finished, bit it is late night ...) I hope this helps, I am really writing this as much to clarify my ideas for myself ... There is a lot of work to do to make this setup mathematically rigorous, although I hope that can be done. If anybody have some interesting relevant references, please chime in ...