Am I understanding differences between Bayesian and frequentist inference correctly?

Question

1. Given a sequence of independent experiments, each having as its outcome either success or failure, the probability of success being some number p between 0 and 1:

A Bayesian would consider the results of the k experiments $X_1, X_2, ..., X_k$ fixed while p would be random, then assume a prior distribution on p and then come up with a posterior distribution for p given the results of the k experiments. Using a distribution for p, a Bayesian would use credible intervals.

A frequentist would consider p a fixed value but the results of the k experiments $X_1, X_2, ..., X_k$ random and would try to come up with a maximum likelihood estimate. A frequentist would use confidence intervals.

2. Hypothesis testing:

A frequentist would use normal approximation to come up with a p-value to see if there is any reason to reject a null hypothesis $H_0$ in favor of an alternative hypothesis $H_1$. For a frequentist, $P(H_0), P(H_1) \in \{0,1\}$

A Bayesian would assign a prior probability to $H_0$ and compute a posterior probability $P(H_0 | k)$. A Bayesian would not absolutely choose between $H_0$ and $H_1$. Rather a Bayesian would be more inclined to $H_0$ if $P(H_0 | k) > 1 - P(H_0 | k) = P(H_1 | k)$

3. Regression: Given regression model $ g(E(Y)) = \beta_0 + \beta_1 X_1 + ... + \beta_k X_k $

A frequentist would consider $\beta_i$'s and other parameters to be fixed and would use MLE or OLS.

A Bayesian would consider $\beta_i$'s and other parameters to be random, assign them prior distributions and then come up with posterior distributions for $\beta_i$'s and other parameters given the y's and X.

---

Anything wrong? Anything I missed?

Answer 1

This questions is too broad, but I thought I would respond to a few points where your statements aren't accurate.

1. Bayesians (typically) believe there is a fixed value for the parameters, but use a probability distribution to represent their uncertainty about what the true value is.

2. A Bayesian is typically interested in the full posterior rather than a point or interval estimate of a particular parameter (although for simplicity in reporting results point or interval estimates are typically provided).

3. A frequentist would not use a normal approximation for hypothesis testing with a point null in a binomial experiment.

4. Even if a frequentist "rejects a null hypothesis" that does not mean they choose the alternative.

5. Bayesians will choose between hypotheses if forced to, but typically we would prefer model averaging.

6. In a regression problem many frequentists use penalized likelihood methods, e.g. lasso, ridge regression, elastic net, etc. and therefore would not be using the MLE or OLS estimators.

Answer 2

A Bayesian would consider the results of the experiments fixed and consider population parameters as stochasts. This in contrast to frequentist, who see the data as "just another sample in an endless stream of samples" and who see the population parameters as fixed (but unknown).

The logical Bayesian order would be: 1. define the prior distribution 2. collect data 3. use that data to update your prior distribution. After updating it is called the posterior distribution.

Mind you that a confidence interval is really different from a credible interval. A confidence interval relates to the sampling procedure. If you would take many samples and calculate a 95% confidence interval for each sample, you'd find that 95% of those intervals contain the population mean.

This is useful to for instance industrial quality departments. Those guys take many samples, and now they have the confidence that most of their estimates will be pretty close to the reality. They know that 95% of their estimates are close, but they can't say that about one specific estimate.

Compare this to rolling dice: if you roll 600 (fair) dice, your best guess is that 1/6, that is 100 dice, will roll a six. But if you someone has rolled 1 die, and asks you: - "What is the probability that this throw was a 6 ?", - the answer "Well, that is 1/6 or 16.6%" is wrong. The die shows either a 6, or some other figure. So the probability is 1, or 0.

When asked before the throw what the probability of throwing a 6 is, a Bayesian would say "1/6" (based on prior information: everybody knows that a die has 6 sides), but a Frequestist would say "No idea" because frequentism is solely based on the data, not on priors.

Likewise, if you have only 1 sample (thus 1 confidence interval), you have no way to say how likely it is that the population mean is in that interval. It is either in it, or not. The probability is either 1, or 0.

---

If a frequentist rejects H0, this means that P(data|H0) is smaller than some threshold. He says "It is very unlikely to find these sort of data if H0 were true, therefore I assume that H0 is not true, thus H1 must be true". Therefore, in this framework, H0 and H1 must be mutually exclusive and cover all possibilities.

As far as I understand, some frequentist say that if H0 is rejected, this does not imply that H1 is formally accepted; others say that rejecting the one equals accepting the other.

Hypothesis testing in a Bayesian method is slightly different. The method is to see how good the data are predicted by Hypothesis A, or B, or C (no need to limit this to 2 hypotheses). The researcher could say: "Hypothesis A explains the data 3 x better than Hypothesis B and 50 times better than Hypothesis C".