# Testing for statistically significant difference between two groups: Frequentist vs Bayesian

When testing for differences between two groups or samples, what is the best approach in concluding statistical significance?

The orthodox approach is to take a t-test or an Anova when there is more than 1 factor to consider.

I've also seen how t-tests can be considered as a linear model and can be generalized. R packages such as BEST or BRMS use a Bayesian approach such as MCMC to obtain parameter estimates. But what can you do after obtaining these estimates to say the two groups are or are not statistically significant?

• A Bayesian technique is common when the mean of the population which you try to estimate is itself distributed according to some distribution which you (approximately) know. For instance you are a wine company that wishes to decide on which plot of grapes to select for the more exclusive bottling. Then you can compare two samples taken for each plot (in order to estimate the mean quality of the plots), but you may possibly include knowledge/information how the quality of the plots is usually distributed. (without such prior information Bayesian techniques are subjective and not exact) Sep 18, 2019 at 15:58

## 3 Answers

When testing for differences between two groups or samples, what is the best approach in concluding statistical significance?

This is completely contextual. If you want guaranteed frequency properties (assuming the assumptions of each procedure are not grossly violated), then I would say a Frequentist approach is best. If you have small data and good prior information about the effects or baseline values, then I would say Bayesian methods are best. Depends on what you want.

The orthodox approach is to take a t-test or an Anova when there is more than 1 factor to consider.

Assuming the outcome is normally distributed. This isn't the case for binary outcomes (though I'm sure you could make a good argument for ANOVA).

But what can you do after obtaining these estimates to say the two groups are or are not statistically significant?

Ah, now this is a different question all together. If you want to evaluate if two groups are not different, you might be interested in a "non-inferiority" test.

A frequentist p-value and a Bayesian posterior tell you different things.

• P-value: $$P(\text{data} | \text{hypothesis})$$ or the probability of data given a hypothesis being true (more precisely a test statistic which is a function of data but that detail doesn't really matter here).
• Posterior: $$P(\text{hypothesis} | \text{data})$$ or the probability of an hypothesis being true given data you've observed.

Whichever is better for your situation depends on what question you're answering! My guess is that most of the time people care about the probability of some hypothesis being true given the data they've seen and so they should look at a Bayesian posterior. This guess is because it's my own experience on what I care about when I do analysis and the fact that when people misinterpret the p-value they usually misinterpret it as a Bayesian posterior (see Am I taking crazy pills here or this A/B testing tool's documentation dead wrong?).

Unfortunately, there is not a clear answer to this question. Bayesian methods and Frequentist methods answer different questions. They are also responding to different criteria and have different capabilities. If the data is normally distributed, with unequal variances, and a flat prior then an analytic form exists for both the Frequentist and the Bayesian method, but the results are not the same.

Using a Frequentist method, you can use Student’s distribution as your test distribution, with the appropriate test statistic. Using a Bayesian approach, you would use the Behrens-Fisher distribution. You have the same data but a different outcome. Understanding that difference in outcome is key to understanding the differences.

Let us start with a simple null hypothesis of $$H_0:\mu_1\ge\mu_2$$ with $$H_A:\mu_1<\mu_2$$. This is a one-sided test. You would calculate your $$t$$ statistic and then find the area under the distribution, usually by looking it up on a table. You are conditioning your result on the sample estimate of the variances. You are also conditioning on the null as if perfectly true. If it is in the rejection region, then your result is statistically significant. Your rejection region would be chosen by choosing a value, $$\alpha$$. If you follow this procedure an infinite number of times then you are guaranteed to be made of a fool of no more than $$\alpha$$ percent of the time.

When using Bayesian thinking, there is no null hypothesis. Without a null to condition upon, the nature of the problem changes. Instead of a null and an alternative, there are just two hypotheses. If your priors are flat as assumed above, then you would calculate your test statistic, though with a different formula, and you would find the area under the curve of the Behrens-Fisher distribution. You would have arrived at that curve by marginalizing out all possible values of $$\sigma_1^2,$$ and $$\sigma_2^2$$. Rather than condition on the point statistic for the sample, a Bayesian method considers all possible values for both parameters and removes them through the law of total probability by summing the values weighted by their densities. That provides a posterior probability for both regions. As there is no $$\alpha$$, there is no concept of “significance.” There is no acceptance or rejection region. There are just posterior probabilities given the data observed.

The fact that identical data does not produce an identical inference should be the warning that you are not doing the same thing with the same data. The Frequentist statistician is asking, “if the null is true, what is the probability of observing a result as extreme or more extreme than the sample?” The Bayesian statistician is asking, “given the data, what is the probability that $$\mu_1\ge\mu_2$$ versus its complement?” The Frequentist solution guarantees an average coverage level, $$\alpha,$$ and also permits you to control for statistical power. The Bayesian method is providing a direct probability statement given the data. However, a p-value of less than five percent does not map to a ninety-five percent Bayesian probability that a hypothesis is true. A p-value of less than five percent might easily be a posterior probability of eighty-three percent or some other value.

The p-value is based on the sample space as being random. The Bayesian posterior is based on the parameter space being treated as random. You cannot use p-value thinking to work out Bayesian inference.

Now let us change the problem slightly. If the null hypothesis were $$H_0:\mu_1=\mu_2$$, then you are conditioning on that as if true when testing your sample. There does not exist a Bayesian counter-part to this hypothesis. The probability a point is true is always zero for continuous data. The area of the hypothesis region is zero, always. This is called a sharp null hypothesis.

Bayesian methods cannot test a sharp null. The only workaround for this problem is to build a region of practical equivalence. The Bayesian hypothesis would be $$|\mu_1-\mu_2|. For example, if you took Indiana US quarters and Florida quarters and flipped them 100,000 times each would you treat a probability of heads of .5 as different from .5000000001? What if you flipped them eight times? A Bayesian method can work on “close enough” but not equal.

Now let us make the problem a bit more specific. Let’s imagine that we are concerned that the Cincinnati Red’s batting average is higher than the Pittsburgh Pirate’s batting average for the 2020 season. Let’s pretend that it is Spring 2020 and we have seen five weeks of games. We have robust prior information. We know the batting averages of all non-rookies for the previous seasons. For rookies, except walk-ons, we have their data from their time in the farm teams, and we have prior knowledge of how rookies change between the minors and the majors.

The Frequentist would have to discard all of this information as it is not part of the sample. The Bayesian would fully incorporate this into the prior. It is known that the Frequentist parameter estimates would be inadmissible statistics. The loss of information would make the Frequentist estimators noisier and the inferences less reliable.

This brings up the question of “why are you asking which team has a higher average?” If you are gambling, then you have to use a Bayesian method because Bayesian methods are coherent. They generate probabilities that can be used to set betting odds. However, if your concern is scientific, then the Frequentist estimators are unbiased while the Bayesian ones are guaranteed to be biased. Furthermore, you do not have to assume that the prior information is good information.

Are your concerns Frequentist concerns or Bayesian ones? Answer that and you have your methodology. If you use a Frequentist method, then you are significant if you are in a rejection region for a given $$\alpha.$$ If you use a Bayesian method, it is significant if you decide it is.