I am analyzing a 2x2 table from a small dataset of 30 patients. We are retrospectively trying to find some variables that give a hint about which treatment to choose. The variables (obs normal / strange) and treatment decision (A/B) are of special interest and therefore the data looks like this:

\begin{array} {|r|r|r|r|} \hline \text{Obs/Tr. Dec.} &\text{A} &\text{B}\\ \hline \text{normal} &12 &13 &25\\ \hline \text{strange} &0 &5 &5\\ \hline &12 &18 &30\\ \hline \hline \end{array}

Obviously one cell lacks on entries which excludes a chi-squared test and Fisher's exact test doesn't give a saturating p-value (but still <10%). So my first idea was to find a test with a greater power and I was reading in a blog and in this article about Barnard's and Boschloos test, that in general there are three scenarios which yield to a powerful test:

  1. Column and Rowsums fixed $\rightarrow$ Fisher's exact test
  2. Column or (xclusive) Rowsums fixed $\rightarrow$ Barnard's exact Test
  3. None are fixed $\rightarrow$ Boschloos's exact Test

The article above pointed out that the sum of treatment A and treatment B are almost never known before, so we can exclude Fisher's exact test. But what about the other alternatives? In case control where we have healthy controls we can control the placebo and verum group which numbers we can control, so one would choose 2: Barnard. In my case I am not sure, because on one the hand we have a similar mathematical problem (sum of observation levels equivalent to sum of placebo / verum), which leads to Barnard but the design is different, because we can't control the nr. of observation normal/strange before taking the sample which leads to 3: Boschloo.

So which test should be used and why? Of course I want high power.

(Another question that I would like to know is, if in case of chisq.test in r it wouldn't be better to use prop.test(x, alternative = "greater")? The theoretical aspects are explained here.)

  • $\begingroup$ Would you have asked this question if Fisher's test would have given a p value below your significance level? $\endgroup$
    – Michael M
    Sep 2 '15 at 21:27
  • $\begingroup$ Since the columns are fixed (it sounds like your article is suggesting Barnard's), but I couldn't get to it without paying :( $\endgroup$
    – MikeP
    Sep 2 '15 at 21:31
  • $\begingroup$ @Michael: I think it's a relevant problem in general, but without the specific problem I might not have considered a deeper research. $\endgroup$
    – Taz
    Sep 2 '15 at 21:56
  • $\begingroup$ @Mike: Sry, I was in institute and didn't think about paywall. If I find a free solution, I will add it. However, I think I didn't point out the problem clear enough. In my case the Treatmentgroups are not controlled, instead they are a consequence of some manual diagnosis by a doctor and I want to find out if the decision for Treatment A or B is related to the Observation variable. And also which test to apply and how to apply it optimal. $\endgroup$
    – Taz
    Sep 2 '15 at 22:12
  • $\begingroup$ Ahhh, so a person entering the study could conceivably have ended up in any of the four categories by the end? $\endgroup$
    – MikeP
    Sep 3 '15 at 12:49

There may be some confusion about term "Barnard"s test or "Boschloo"s test. Barnard's exact test is an unconditional test in the sense that it does not condition on both margins. Therefore, both the second and third bullets are Barnard's test. We should instead write:

  1. Both margins fixed (Hypergeometric Dist'n)→Fisher's exact test
  2. One margin fixed (Double Binomial Dist'n)→Barnard's exact test
  3. No margins fixed (Multinomial Dist'n)→Barnard's exact test

Barnard's exact test encompasses two types of tables, so we distinguish the two by saying "binomial" or "multinomial" model as appropriate.

Typically, Barnard's exact test either uses a Z-pooled (aka Score) statistic to determine the 'as or more extreme' tables. Note the original Barnard paper (1947) uses a more complicated approach to determine the more extreme tables (referred as "CSM"). Boschloo's exact test uses Fisher's p-value to determine the 'as or more extreme' tables. Boschloo's test is uniformly more powerful than Fisher's exact test.

For your dataset, it sounds like neither margins were fixed, so would recommend using Boschloo's exact test with a multinomial model. I found Boschloo's test slightly better for unbalanced margin ratios (although typically very similar to Barnard's exact test with Z-pooled statistic). However, since both Boschloo's test and multinomial models are much more computationally intensive, you can also use the binomial model (the reasoning for why this would still be appropriate is a little complicated; to briefly summarize, the margins are an approximately ancillary statistic, so it's alright to condition on margin). For more details on the exact tests and information on implementation, please use the Exact R package (https://cran.r-project.org/web/packages/Exact/Exact.pdf). I am the author of the package and it's a more updated version of the code on the blog.

  • 1
    $\begingroup$ Thanks for your clear statement! Very nice to have this explanation in a few lines. In the end I did it like you wrote after reading the paper which is very good, but also very long;-) $\endgroup$
    – Taz
    Oct 1 '15 at 18:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.