4
$\begingroup$

I am currently studying the textbook In All Likelihood -- Statistical Modelling and Inference Using Likelihood by Yudi Pawitan. Section Inverse probability: the Bayesians of chapter 1 says the following:

The first modern method to assimilate observed data for quantitative inductive reasoning was published (posthumously) in 1763 by Bayes with his Essay towards Solving a Problem in the Doctrine of Chances. He used an inverse probability, via the now-standard Bayes theorem, to estimate a binomial probability. The simplest form of the Bayes theorem for two events $A$ and $B$ is

$$P(A \vert B) = \dfrac{P(AB)}{P(B)} = \dfrac{P(B \vert A)P(A)}{P(B \vert A)P(A) + P(B \vert \overline{A})P(\overline{A})}. \tag{1.1}$$

Suppose the unknown binomial probability is $\theta$ and the observed number of successes in $n$ independent trials is $x$. Then, in modern notation, Bayes's solution is

$$f(\theta \vert x) = \dfrac{f(x, \theta)}{f(x)} = \dfrac{f(x \vert \theta) f(\theta)}{\int f(x \vert \theta) f(\theta) d \theta}, \tag{1.2}$$

where $f(\theta \vert x)$ is the conditional density of $\theta$ given $x$, $f(\theta)$ is the so-called prior density of $\theta$ and $f(x)$ is the marginal probability of $x$. (Note that we have used the symbol $f(\cdot)$ as a generic function, much like the way we use $P(\cdot)$ for probability. The named argument(s) of the function determines what the function is. Thus, $f(\theta, x)$ is the joint density of $\theta$ and $x$, $f(x \vert \theta)$ is the conditional density of $x$ given $\theta$, etc.)

Leaving aside the problem of specifying $f(\theta)$, Bayes had accomplished a giant step: he had put the problem of inductive inference (i.e. learning from data $x$) within the clean deductive steps of mathematics. Alas, 'the problem of specifying $f(\theta)$' a priori is an equally giant point of controversy up to the present day.

There is nothing controversial about the Bayes theorem (1.1), but (1.2) is a different matter. Both $A$ and $B$ in (1.1) are random events, while in the Bayesian use of (1.2) only $x$ needs to be a random outcome; in a typical binomial experiment $\theta$ is an unknown fixed parameter. Bayes was well aware of this problem, which he overcame by considering that $\theta$ was generated in an auxiliary physical experiment - throwing a ball on a level square table - such that $\theta$ is expected to be uniform in the interval $(0, 1)$. Specifically, in this case we have $f(\theta) = 1$ and

$$f(\theta \vert x) = \dfrac{\theta^x(1 - \theta)^{n - x}}{\int_0^1 u^x(1 - u)^{n - x} du} \tag{1.3}$$

Fisher was very respectful of Bayes's seeming apprehension about using an axiomatic prior; in fact, he used Bayes's auxiliary experiment to indicate that Bayes was not a Bayesian in the modern sense. If $\theta$ is a random variable then there is nothing 'Bayesian' in the use of the Bayes theorem. Frequentists do use the Bayes theorem in applications that call for it.

It is this last part that I am curious about:

Fisher was very respectful of Bayes's seeming apprehension about using an axiomatic prior; in fact, he used Bayes's auxiliary experiment to indicate that Bayes was not a Bayesian in the modern sense. If $\theta$ is a random variable then there is nothing 'Bayesian' in the use of the Bayes theorem. Frequentists do use the Bayes theorem in applications that call for it.

So what specifically is the controversy here? Why would considering $\theta$ to be a random variable not be 'Bayesian'?

Based on prior (hehe, no pun intended) study, I understanding that there is significant controversy here between Bayesians and Frequentists. And I understand the differences between the two philosophies. But it's not clear to me what the specific problem is with Bayes's work (or, rather, what it is that Frequentists find so controversial about his work).

I would greatly appreciate it if people would please take the time to explain this.

$\endgroup$
10
  • 7
    $\begingroup$ Essentially, if the prior distribution on $\theta$ is provided by the experimental setting, ie if there is no debate about the prior, then this is not a Bayesian framework but rather an error-in-variable type model. $\endgroup$ – Xi'an Feb 19 '20 at 18:40
  • 4
    $\begingroup$ Frequentists find nothing at all controversial about Bayes' Theorem--it is, after all, a simple mathematical lemma. Many frequentist results are proven using Bayes' Theorem and Bayesian approaches. The heart of the matter--beyond a fundamental difference in interpreting the meaning of "probability"--concerns what it means to model parameters by means of probability distributions and how to determine those distributions. $\endgroup$ – whuber Feb 19 '20 at 19:03
  • $\begingroup$ @whuber is it fair to say the "controversy" arises in the Bayesian interpretation of probability, which allows one to use Bayes theorem to assign probability to retrospective events? $\endgroup$ – AdamO Feb 19 '20 at 19:59
  • $\begingroup$ @AdamO I don't think any controversy pertains to the temporal sequence of events. $\endgroup$ – whuber Feb 19 '20 at 20:03
  • 1
    $\begingroup$ There's "subjective" and "objective Bayes", and more than one flavour of each. It's not appropriate to talk about "Bayesians" as if this was a homogeneous group. $\endgroup$ – Lewian Feb 21 '20 at 10:29
3
$\begingroup$

The quote does not say that if $\theta$ is a random variable, this is not Bayesian scenario. It says that the fact that $\theta$ is a random variable doesn't make it a Bayesian setting by itself. As already noticed in the comments, Bayes theorem is just a theorem in probability theory that lets us get the "reversed" conditional probability from $p(a|b)$ to $p(b|a)$. As mentioned here, or here, $a$ and $b$ can be any random variables for Bayes theorem to hold.

In Bayesian statistics we use Bayes theorem with particular kind of random variables, the ones that are "made up", the prior distributions. When Bayesian wants to estimate the distribution of the parameter of interest $\theta$ for the likelihood function $p(x|\theta)$, she uses a prior distribution $p(\theta)$ (the distribution that she assumes that the parameter can follow) and applies Bayes theorem, to calculate the posterior

$$ p(\theta|x) = \frac{p(x|\theta)\;p(\theta)}{p(x)} $$

In non-Bayesian setting, you could use some random variables, in this example, they show how can we calculate the probability of having a cancer given the positive test result $p(C|+)$ by applying Bayes theorem to cancer prevalence $p(C)$ and the probability of the test of giving positive results $p(+|C)$:

$$ p(C|+) = \frac{p(+|C)\,p(C)}{p(+)} $$

In here, both probabilities are known to us, contrary to Bayesian setting where the prior is assumed.

$\endgroup$
5
  • $\begingroup$ I don't understand what you mean by "It says that $\theta$ could be a random variable while this weren't a Bayesian setting." Can you please elaborate? $\endgroup$ – The Pointer Feb 20 '20 at 2:30
  • $\begingroup$ @ThePointer see my edit. $\endgroup$ – Tim Feb 20 '20 at 10:04
  • $\begingroup$ "In here, both probabilities are known to us, contrary to Bayesian setting where the prior is assumed." Is the Bayesian setting really defined as 'when the prior is assumed instead of known'? $\endgroup$ – Sextus Empiricus Feb 22 '20 at 17:03
  • $\begingroup$ @SextusEmpiricus what I wanted to say is that Bayesian statistics is about updating your knowledge or assumptions. However I agree, this is a vague statement. $\endgroup$ – Tim Feb 22 '20 at 18:19
  • $\begingroup$ @Tim, I believe that the cancer example is also about updating knowledge/assumptions. But it is all indeed very vague. I guess that the philosophical problems are more a matter whether it is appropiate to 'treat a parameter that is fixed, e.g. a physical constant, as a variable'. And there is the other (practical) matter about how to 'choose' a prior and the possibility that the method introduces (too much) subjectivity. I guess that it is more about the philosophical matter when it is about the situation whether we consider $\theta $ as a random variable or not. $\endgroup$ – Sextus Empiricus Feb 22 '20 at 19:44
2
$\begingroup$

I think that the confusion stems from a somewhat ambiguous use of the terminology. In my view the term frequentism should be used for an interpretation of probabilities, namely that they refer to data generating processes in reality and correspond to limits of relative frequencies under idealised infinite repetition. In frequentism often parametric models are used in which the parameter is fixed. However, it is also possible to have a process in which a parameter value itself is the result of a potentially repeatable experiment, in which case there can be a frequentist distribution over the parameter, and Bayes theorem can be applied. Otherwise, though, most frequentists would not accept a probability distribution over their parameters, because they are fixed and not random in reality, or rather in what they think of as reality.

What is today called "Bayesian" is an approach in which a prior distribution over the parameter of parametric model is assumed, which normally is interpreted as encoding epistemic probabilities, referring to prior knowledge of the parameter, but not to a real physical process generating it. This approach always requires Bayes theorem. However this can still be connected with a frequentist idea of probabilities, see e.g. the section on "falsificationist Bayes" in Gelman and Hennig "Beyond subjective and objective in statistics".

Furthermore, Fisher favoured so-called "fiducial probabilities", which are epistemic probabilities over the parameters of frequentist models. These are widely rejected these days though. Anyway, in this setting the $\theta$ was treated as random variable, but without prior, not requiring Bayes theorem.

What I find problematic and potentially confusing about the terminology is that seemingly today many would no longer call the use of Bayesian computation with frequentist probabilities Bayesian, despite the fact that Bayes himself did such a thing.

$\endgroup$
1
  • $\begingroup$ Interesting. Thanks for the clarification. $\endgroup$ – The Pointer Feb 21 '20 at 14:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.