Why does the contigency table Bayes Factor test does not behave like an omnibus test?

Question

I have a contigency table and want to know whether the distributions differ between columns. For example, let's say I want to know whether the distribution of children with academic vs. non-academic parents differs between different schools. First, I have collected data at two schools and perform a contigency table Bayes Factor test that shows strong evidence for different distributions between these two schools.

library(BayesFactor)

ctable <- matrix(c(108, 72,
                   159, 21),
                 ncol=2,
                 byrow=T,
                 dimnames=list(school=LETTERS[1:2], parents=c("academic", "non-academic")))

contingencyTableBF(ctable, sampleType="indepMulti", fixedMargin="rows" )

Output:

Bayes factor analysis
--------------
[1] Non-indep. (a=1) : 33542933 ±0%

Against denominator:
  Null, independence, a = 1 
---
Bayes factor type: BFcontingencyTable, independent multinomial

However, if I add further rows to the contigency table (so after continuing data collections at more schools), the evidence for different distributions between schools vanishes.

ctable_ext <- matrix(c(108, 72,
                       159, 21,
                       136, 44,
                       129, 51,
                       142, 38,
                       129, 51,
                       143, 37,
                       136, 44,
                       145, 35,
                       138, 42,
                       139, 41),
                     ncol=2,
                     byrow=T,
                     dimnames=list(school=LETTERS[1:11], parents=c("academic", "non-academic")))

contingencyTableBF(ctable_ext, sampleType="indepMulti", fixedMargin="rows" )

Output:

Bayes factor analysis
--------------
[1] Non-indep. (a=1) : 0.7131625 ±0%

Against denominator:
  Null, independence, a = 1 
---
Bayes factor type: BFcontingencyTable, independent multinomial

I am surprised by this finding as I would have expected the Bayes Factor test to behave like an omnibus test (as it is the case with its frequentist counterpart, the chi-squared test). But then adding additional rows should not affect our conclusion, as the distribution between the first two schools still differs. Or am I missing something here? Can somebody explain why adding more data decreases the evidence?

Answer 1

The Wikipedia page on Bayes' Factor has a nice example using a binomial proportion. This example indicates that the Bayes' factor does incorporate prior beliefs and, depending on these beliefs, can produce a "non-significant" result where a frequentist approach would produce a "significant" result. A uniform prior does not guarantee the Bayes' factor to be in agreement with the frequentist likelihood ratio test. Perhaps this is what is happening in your case.

In your first data set

$\text{Academic 1: }108 (40.4\%) \text{ } 159 (59.6\%)$

$\text{Academic 2: }\hspace{2.5mm}72 (77.4\%) \text{ }\hspace{2.5mm} 21 (22.9\%)$

In your second data set

$\text{Academic 1: }108 (7.2\%) \text{ } 159 (10.6\%)... 139 (9.2\%)$ $\text{Academic 2: }72 (15.1\%) \text{ } \hspace{4mm} 21 (4.4\%)...\hspace{2mm}41 (8.6\%)$

In the second data set most of the proportions are close to 8%. Perhaps the likelihood is not strong enough to overcome the prior.