# Bayes Factors and Type I error Rate

The WIKIPEDIA Page on Bayes Factors, quickly mentions something that caught my attention. The exact quote from WIKIPEDIA is pasted below this post.

Specifically, I'm wondering if there is any property attributed to bayes factors that has made the Wikipedia page to say that $Bayes~Factors$ are known to have lower Type I error rate?

From WIKIPEDIA:

"$M_2$ [i.e., the alternative model] is a more complex model than $M_1$ [i.e., the Null model] because it has a free parameter which allows it to model the data more closely. The ability of Bayes factors to take this into account is a reason why Bayesian inference has been put forward as a theoretical justification for and generalisation of Occam's razor, reducing Type I errors."

## Some thoughts and simulation:

Let me do a quick simulation (and please correct me if I'm wrong anywhere) to see what is exactly meant by Type I error when using bayes factors to do hypothesis testing. Bayes Factors are often benchmarked against cut-off values put forward by Harold Jefferey (also see below 1961, p.432)

So, to attribute Type I error rate to bayes factors, one has to invoke Jeffeys benchamrks. Specifically, for a $Alternative/Null$ Bayes Factor when Null is True we may consider the proportion of the times that we obtain a Bayes factor that is larger than 3 (General Rule). Yet a more fine-grained view is shown below using R:

if(!require(BayesFactor)){install.packages('BayesFactor')}
library(BayesFactor)

n = 1e2
n.sim = 1e3
BF10 = numeric(n.sim)
set.seed(0)

for(i in 1:n.sim){
x = rnorm(n = n, mean = .0, sd = 1)
y = rnorm(n = n, mean = .0, sd = 1)
t.value = unname(t.test(x, y)[])
BF10[i] = unname(ttest.tstat(t.value, n, n, rscale = sqrt(2)/2, simple = TRUE)[])
}


It seems if we take the decision rule of BF larger than 3 is a Type I error, than in our case Type I error rate is $1.3$%.

## Summarize the erroneous support for alternative when NULL is True (Type I error rate):

## General rule:
cat("Erroneously Reject ",(length(BF10[3 < BF10])/n.sim)*1e2,"%")

## Fine-grained rule:
cat("Anecdotally Reject", (length(BF10[1 < BF10 & BF10 <= 3])/n.sim)*1e2,"%")
cat("Substantially reject", (length(BF10[3 < BF10 & BF10 <= 10])/n.sim)*1e2, "%")
cat("Strongly reject", (length(BF10[10 < BF10 & BF10 <= 30])/n.sim)*1e2,"%")
cat("Decisively reject", (length(BF10[BF10 >= 100])/n.sim)*1e2,"%")


# Here are Jeffereys' benchmarks: • The rough equivalence of a Bayes factor to a significance test would be B=19. You don't have an explicit cost function to relate type I and type II errors. A Bayes factor of 3 is roughly equivalent to p<.25. I say roughly because they don't mean the same thing and the posterior is an actual probability distribution and a sampling test is a worst case distribution. – Dave Harris Jun 19 '17 at 21:32
• I would really have to think about where the equivalences might be at because Bayesian posteriors provide you the probability some hypothesis is true. Frequentist probability statements are about the probability of observing data as or more extreme, given the null is true. – Dave Harris Jun 19 '17 at 21:35
• @DaveHarris, I'm interested (even though my field is not stats), could you please expand on what I just added? – rnorouzian Jun 19 '17 at 21:35
• Let us imagine a situation where the Pr(H1|x)=.51 and Pr(H2|x)=.49, but someone puts a gun to your head and says choose the correct brand of toothpaste or I will kill you right now. Type I and II error takes on a different meaning because of the cost function. You would choose H1. This would still be true at Pr(H1|x)=.99. Having a one percent chance of dying today is a massive risk compared to the background risk. – Dave Harris Jun 19 '17 at 21:37
• The problem is similar to the setting of $\alpha$ in Pearson-Neyman methods. For any binary hypothesis, as long as one is greater than the other and you select it then you have decided that the information collected was sufficient to make a decision. – Dave Harris Jun 19 '17 at 21:39

## 1 Answer

Oh, how I dislike the language used in the Wikipedia article. Bayes factors can have problems that do not exist for posterior probabilities, even though Bayes factors could be seen as a function of posterior probabilities. Bayesian methods never use less information than non-Bayesian methods, but they may use as much and may have precisely the same outcome under very careful sets of assumptions.

I am going to try and answer this in light of three papers and a book. They are:

Jeffreys, William and Berger, James. Sharpening Ockham's Razor On A Bayesian Strop. Technical Report 91-44C. Purdue University

Wetzels, Ruud, et al. Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t-tests. Perspectives on Psychological Science. 6(3) 291–298

The third is by Abraham Wald that discusses his complete class theorem, but I am not at a place where I can look up a bibliography for it.

The fourth is a book:

COX, R. T., The Algebra of Probable Inference, Johns Hopkins University Press, Baltimore, MD, (1961).

The first thing to understand is that from the Complete Class Theorem, Bayesian methods cannot be stochastically dominated, but the Bayesian method can stochastically dominate a non-Bayesian method. A Bayesian method can never be less accurate, but it can be more accurate. This alone can reduce type I errors. The difference is generally in the presence of prior information. In the presence of real prior knowledge, a Bayesian solution will dominate all others. It is rare not to have some real knowledge. While there may be a desire to feign ignorance so as to minimize bias, it is rare to be truly ignorant.

The second issue comes from Wetzels' paper. They compare 855 published t-test to their Bayesian result and found that in many cases the Bayesian solution was not significant or even supported the null. The added structure of non-Bayesian methods makes it more likely that a finding will be seen as significant. While, of course, it isn't known which hypotheses are really true and false, the impact is systemic and not idiosyncratic. Given the high false discovery rate in psychology found in other papers, it would tend to support the Bayesian method.

Wetzels found that simply using stricter rules would not help in setting a cutoff for $\alpha$. The math did not improve because higher cutoffs trigger higher sample sizes for power and this just shifted the problem, but did not solve it.

Finally, axiom systems such as Cox's assign probabilities to each hypothesis, something that non-Bayesian methods do not do. Indeed, no number is assigned to any hypothesis, just the to the probability of seeing the data if the null is true. In Frequentist methods, the acceptance or rejection of the null is an all-or-nothing situation. The cut-off is arbitrary. There is no similar situation forced in Bayesian methods. This allows more information into the Bayesian method by marginalizing over the parameter space as the exact parameter values are not of interest. Non-Bayesian methods only consider the point expectation based on the data.

If type I error exists in a Frequentist model, then a sample statistic is as extreme or more extreme than would be expected under the null to some degree of confidence. If that same statistic exists in a Bayesian model, there may be some of the posterior density that lies outside the false alternative. Likewise, other parameters in the model may be influencing the hypothesis of the parameter of interest. It may be that under some density locations of the other parameters, the false alternative would not be taken. Bayes factors average over the nuisance parameter space, instead of using point estimates. This gets more information in the Bayes factor than the Frequentist solution, or at least can get more in it.

The likelihood ratio can be thought of as the ratio of the best case for one hypothesis versus the best case for the other hypothesis. The Bayes factor is the average density in one region versus the average density in another region. The Bayes factor will tend to be more conservative in rejecting a true null.

• This doesn't seem to answer the question. – Jake Westfall Jun 19 '17 at 3:03
• Dear Dave, thanks for your response could please be more specific about the Type I error rate and the quote I am referring to in the Wikipedia? – rnorouzian Jun 19 '17 at 3:27