Why would considering $\theta$ to be a random variable not be 'Bayesian'?

Question

I am currently studying the textbook In All Likelihood -- Statistical Modelling and Inference Using Likelihood by Yudi Pawitan. Section Inverse probability: the Bayesians of chapter 1 says the following:

The first modern method to assimilate observed data for quantitative inductive reasoning was published (posthumously) in 1763 by Bayes with his Essay towards Solving a Problem in the Doctrine of Chances. He used an inverse probability, via the now-standard Bayes theorem, to estimate a binomial probability. The simplest form of the Bayes theorem for two events $A$ and $B$ is

$$P(A \vert B) = \dfrac{P(AB)}{P(B)} = \dfrac{P(B \vert A)P(A)}{P(B \vert A)P(A) + P(B \vert \overline{A})P(\overline{A})}. \tag{1.1}$$

Suppose the unknown binomial probability is $\theta$ and the observed number of successes in $n$ independent trials is $x$. Then, in modern notation, Bayes's solution is

$$f(\theta \vert x) = \dfrac{f(x, \theta)}{f(x)} = \dfrac{f(x \vert \theta) f(\theta)}{\int f(x \vert \theta) f(\theta) d \theta}, \tag{1.2}$$

where $f(\theta \vert x)$ is the conditional density of $\theta$ given $x$, $f(\theta)$ is the so-called prior density of $\theta$ and $f(x)$ is the marginal probability of $x$. (Note that we have used the symbol $f(\cdot)$ as a generic function, much like the way we use $P(\cdot)$ for probability. The named argument(s) of the function determines what the function is. Thus, $f(\theta, x)$ is the joint density of $\theta$ and $x$, $f(x \vert \theta)$ is the conditional density of $x$ given $\theta$, etc.)

Leaving aside the problem of specifying $f(\theta)$, Bayes had accomplished a giant step: he had put the problem of inductive inference (i.e. learning from data $x$) within the clean deductive steps of mathematics. Alas, 'the problem of specifying $f(\theta)$' a priori is an equally giant point of controversy up to the present day.

There is nothing controversial about the Bayes theorem (1.1), but (1.2) is a different matter. Both $A$ and $B$ in (1.1) are random events, while in the Bayesian use of (1.2) only $x$ needs to be a random outcome; in a typical binomial experiment $\theta$ is an unknown fixed parameter. Bayes was well aware of this problem, which he overcame by considering that $\theta$ was generated in an auxiliary physical experiment - throwing a ball on a level square table - such that $\theta$ is expected to be uniform in the interval $(0, 1)$. Specifically, in this case we have $f(\theta) = 1$ and

$$f(\theta \vert x) = \dfrac{\theta^x(1 - \theta)^{n - x}}{\int_0^1 u^x(1 - u)^{n - x} du} \tag{1.3}$$

Fisher was very respectful of Bayes's seeming apprehension about using an axiomatic prior; in fact, he used Bayes's auxiliary experiment to indicate that Bayes was not a Bayesian in the modern sense. If $\theta$ is a random variable then there is nothing 'Bayesian' in the use of the Bayes theorem. Frequentists do use the Bayes theorem in applications that call for it.

It is this last part that I am curious about:

Fisher was very respectful of Bayes's seeming apprehension about using an axiomatic prior; in fact, he used Bayes's auxiliary experiment to indicate that Bayes was not a Bayesian in the modern sense. If $\theta$ is a random variable then there is nothing 'Bayesian' in the use of the Bayes theorem. Frequentists do use the Bayes theorem in applications that call for it.

So what specifically is the controversy here? Why would considering $\theta$ to be a random variable not be 'Bayesian'?

Based on prior (hehe, no pun intended) study, I understanding that there is significant controversy here between Bayesians and Frequentists. And I understand the differences between the two philosophies. But it's not clear to me what the specific problem is with Bayes's work (or, rather, what it is that Frequentists find so controversial about his work).

I would greatly appreciate it if people would please take the time to explain this.

Answer 1

The quote does not say that if $\theta$ is a random variable, this is not Bayesian scenario. It says that the fact that $\theta$ is a random variable doesn't make it a Bayesian setting by itself. As already noticed in the comments, Bayes theorem is just a theorem in probability theory that lets us get the "reversed" conditional probability from $p(a|b)$ to $p(b|a)$. As mentioned here, or here, $a$ and $b$ can be any random variables for Bayes theorem to hold.

In Bayesian statistics we use Bayes theorem with particular kind of random variables, the ones that are "made up", the prior distributions. When Bayesian wants to estimate the distribution of the parameter of interest $\theta$ for the likelihood function $p(x|\theta)$, she uses a prior distribution $p(\theta)$ (the distribution that she assumes that the parameter can follow) and applies Bayes theorem, to calculate the posterior

$$ p(\theta|x) = \frac{p(x|\theta)\;p(\theta)}{p(x)} $$

In non-Bayesian setting, you could use some random variables, in this example, they show how can we calculate the probability of having a cancer given the positive test result $p(C|+)$ by applying Bayes theorem to cancer prevalence $p(C)$ and the probability of the test of giving positive results $p(+|C)$:

$$ p(C|+) = \frac{p(+|C)\,p(C)}{p(+)} $$

In here, both probabilities are known to us, contrary to Bayesian setting where the prior is assumed.

Answer 2

I think that the confusion stems from a somewhat ambiguous use of the terminology. In my view the term frequentism should be used for an interpretation of probabilities, namely that they refer to data generating processes in reality and correspond to limits of relative frequencies under idealised infinite repetition. In frequentism often parametric models are used in which the parameter is fixed. However, it is also possible to have a process in which a parameter value itself is the result of a potentially repeatable experiment, in which case there can be a frequentist distribution over the parameter, and Bayes theorem can be applied. Otherwise, though, most frequentists would not accept a probability distribution over their parameters, because they are fixed and not random in reality, or rather in what they think of as reality.

What is today called "Bayesian" is an approach in which a prior distribution over the parameter of parametric model is assumed, which normally is interpreted as encoding epistemic probabilities, referring to prior knowledge of the parameter, but not to a real physical process generating it. This approach always requires Bayes theorem. However this can still be connected with a frequentist idea of probabilities, see e.g. the section on "falsificationist Bayes" in Gelman and Hennig "Beyond subjective and objective in statistics".

Furthermore, Fisher favoured so-called "fiducial probabilities", which are epistemic probabilities over the parameters of frequentist models. These are widely rejected these days though. Anyway, in this setting the $\theta$ was treated as random variable, but without prior, not requiring Bayes theorem.

What I find problematic and potentially confusing about the terminology is that seemingly today many would no longer call the use of Bayesian computation with frequentist probabilities Bayesian, despite the fact that Bayes himself did such a thing.