2
$\begingroup$

I'm perplexed by the comment of a reviewer, who has taken issue with the sample size from a study I conducted. He says, relatively to the 2X2 contingency table below and a (significant) 2-sample test for equality of proportions:

9 and 19 are too low, all cells should have 30 observations minimum, no statistical test will fix that.

Here is the contingency table in question:

Gender characteristic X absence of characteristic X
Men 19 194
Women 9 251

The hypothesis here was that there is a difference of proportions between men and women relatively to the characteristic of interest (i.e. is the % of women with characteristic X different from the % of men with this characteristic?). If it's in any way relevant, the population studied here is of about 25,000 individuals.

A two-sided test of proportion (in R, prop.test(c(9,19), c(260, 213))) gives a p-value of 0.02105. (As a side note, a Fisher test gives a p-value = 0.01751.)

In my discipline and in the journal where I intend to publish, p-values < 0.05 are usually deemed as significant; the reviewer doesn't seem to take issue with this convention, so at first glance it is not a problem relative to significance thresholds and their relevance.

His point is not about model misspecification either, as the rationale for using only these two variables (gender/outcome) is given in the paper and he did not object to it.

His point seems to be that any given sample should follow this rule of "30 observed counts minimum by cell" in order to infer anything reliably.

I never read such a requirement before, and it is something I don't quite understand: in this case, even with a huge sample size, you may fail to satisfy this rule if an event is very rare in one of the groups you compare.

Now I still try to make sense of the reviewer's comment, in order to answer it or to understand a potential mistake I made here. It was my understanding that the sample size has no or limited impact on type-1 errors (see Can a small sample size cause type 1 error?). The reviewer insists, and makes me wonder if I'm somehow mistaken, if I misread something, if I conducted an incorrect test, or something else.

So, basically, I'm looking for a second opinion. Is there something incorrect in my analysis here? If not, do you know some reliable academic references that I can use to reinforce my answer to his comment? And if I'm mistaken, I'd be of course more than interested by an explanation.

Thanks,

$\endgroup$

1 Answer 1

4
$\begingroup$

The reviewer's "rule" of 30-per-cell is obviously outdated. I think their bigger point is that your samples are small (true) and you're likely under powered.

In R:

 pwr::pwr.2p2n.test(h = pwr::ES.h(p1 = (19/194),
                                  p2 = (9/251)),
                    n1 = (19+194),
                    n2 = (9+251),
                    sig.level = .05)

     difference of proportion power calculation for binomial distribution (arcsine transformation) 

              h = 0.2555792
             n1 = 213
             n2 = 260
      sig.level = 0.05
          power = 0.7897418
    alternative = two.sided

NOTE: different sample sizes

You don't quite have even 80% power to detect this effect - that is, it just so happens to be the smallest effect you're powered to detect with this sample. Given that small samples are likely to over-estimate effect sizes, it's quite likely that the true effect in the population is smaller than this. And if that's the case, then you're under powered, and I'd be worried about this being a false-positive.

It can often be sufficient to appropriately caveat results throughout a manuscript.

$\endgroup$
7
  • 4
    $\begingroup$ +1. "Outdated" is a kind charaterization, because it's difficult to imagine any date at which this was a reasonable rule of thumb. $\endgroup$
    – whuber
    Feb 3, 2023 at 18:22
  • 1
    $\begingroup$ I like the point about the power (though I see no evidence from their comment that power is what the reviewer had in mind). A common advice is to not calculate post-hoc power but here it's only used as a cautionary tale rather than as an important part of the analysis. Also, these samples are are only "small" because the probabilities are small. $\endgroup$
    – dipetkov
    Feb 3, 2023 at 21:00
  • 2
    $\begingroup$ In my experience, any of the "rules" people have about sample size come from incorrect assumptions about how power works. Perhaps I'm incorrect. I agree re post-hoc power, but note that I did not condone using said analysis to argue that they are powered to detect an effect. Instead I used it to highlight that their effect is right on the edge of what can detected with 80% power, which I think is extremely unlikely unless your effect estimate is inflated. $\endgroup$
    – David B
    Feb 3, 2023 at 21:01
  • 1
    $\begingroup$ Now it occurs to me that there might be more to this story. As the "minimum 30" is so much higher than even the most conservative standard recommendations, the reviewer actually might have a lot of experience with the trait being analyzed. So they know the trait is rare and even may have a good guess at how rare. The number 30 might very well come from that. $\endgroup$
    – dipetkov
    Feb 3, 2023 at 21:32
  • 2
    $\begingroup$ Maybe? 40 per cell for an ANOVA is a 10-year old rule-of-thumb in some psychology sub-fields, so 30 didn't strike me as particularly noteworthy. $\endgroup$
    – David B
    Feb 3, 2023 at 21:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.