Comparison of bayesian and frequentist methods example

Question

I am trying to understand the intuition behind Bayesian and frequentist hypothesis testing and came up with the following example to understand the differences. Do you agree that the following reasoning and intuition is correct? Especially the intuitive reason for needing a prior.

Suppose that a covid test has 95% probability of being negative given that the patient is negative and 5% probability of testing positive given that he is negative. Suppose that the same test has a 60% probability of testing positive given that the patient is positive and 40% probability of testing negative given that he is positive. In summary:

$P(\text{tested negative} \vert \text{is negative})=0.95$

$P(\text{tested positive} \vert \text{is negative})=0.05$

$P(\text{tested negative} \vert \text{is positive})=0.40$

$P(\text{tested positive} \vert \text{is positive})=0.60$

Now suppose I want evidence to support the claim that the patient is positive. Suppose the patient tested positive.

Bayesian perspective:

From a Bayesian perspective, the evidence would be in the form of a probability, namely the probability that the patient is indeed positive given that he tested positive = $P(\text{is positive} \vert \text{tested positive})$.

As there are two possibilities:

1. The patient tested positive and is positive
2. The patient tested positive and is negative

It seems logical that the probability that the patient is positive given that he tested positive, is given by the quotient:

$\frac{P(\text{tested positive} \vert \text{is positive})}{P(\text{tested positive} \vert \text{is positive}) + P(\text{tested negative} \vert \text{is negative})}$ = $\frac{0.05}{0.05 + 0.95}$ = $0.05$

However, this calculation cannot be the correct probability, as it does not take into account the frequencies of being positive and being negative. For example in a country with no covid, the probability being positive should be 0 and not 0.05. Therefore we must multiply the 0.05 by the prior probability of being positive. As we do not know this probability we must set it subjectively.

Frequentist perspective:

From a frequentist perspective, if we want to provide evidence for the claim that the patient is indeed positive then we assume the opposite and assess how unlikely/suprising this result is. Hence, we want to find the probability that the patient is positive given that he is negative = $P(\text{tested positive} \vert \text{is negative})=0.05$. We cannot say that the patient is 5% likely to be positive, but we can say that assuming if he were negative, such a result would only occur 5% of the time. Therefore, if we consider 5% to be suprising enough (95% threshold), we will accept this a strong evidence for the hypothesis that the patient is positive.

Answer 1

It's not the best example to understand both approaches. In your example, you know all the probabilities and the answer can be given directly from this knowledge, there's nothing Bayesian or frequentist about it. Using Bayes theorem for calculation doesn't make it Bayesian, as it is a general theorem about probabilities.

Bayesians vs frequentists differ in how they learn from the data. What Bayesians do differently is they assume priors for the distributions of the unknown parameters and use the Bayes theorem to update the priors using the data. Frequentists consider only the data. The other difference is how they think of probabilities, as a subjective measure of belief in something vs a statement about long run frequencies. So they would differ in estimating the probability from the data and interpreting it.

Answer 2

I think the example you picked is not going to serve well to pin down the difference between the bayesian and frequentist approach in regards to hypothesis testing, because hypothesis testing is already done by the researchers to create the information below:

𝑃(tested negative|is negative)=0.95  
𝑃(tested positive|is negative)=0.05  
𝑃(tested negative|is positive)=0.40  
𝑃(tested positive|is positive)=0.60

When you have this model of the world - a probability distribution that applies to a population-, regardless of a frequentist or bayesian approach, we can apply the probability theorem and the Bayes' rule, which is a fundamental mathematical fact. There is no need to do hypothesis testing because you already have the model at hand.

I would recommend to look at A/B testing with proportions - a hypothesis test to come up with probabilities for 2 groups and decide whether the difference for these proportions is significant enough to use it as a model -. It should be relatively easy to find hands-on examples from both schools of thought. I hope this helps.