5
$\begingroup$

Among statisticians there is an ongoing debate whether the long-standing "standard" threshold for declaring the outcome of a statistical test as "significant" (a.k.a. alpha-level) should be reset from the current default value of p<0.05 to p<0.005. This suggestion is floating around for a while but has ben specifically expressed in a paper published some years ago: Benjamin, et al. "Redefine statistical significance." Nature human behaviour 2.1 (2018): 6-10. https://doi.org/10.1038/s41562-017-0189-z

If I want to do this without losing too much power, this will require that I adjust my sample size accordingly. My question is: For the simplest case of a two-sample t-test and a current sample size X (that is deemed sufficient to detect an effect with alpha=0.05), is there a way to say in general by how much I have to increase my sample size if I switch to alpha=0.005? Specifically --is there a way to say that, that does not require making a power-analysis for a specific case?

$\endgroup$
1
  • $\begingroup$ While we agree that statistical significance is a poor gatekeeping procedure for science, merely suggesting to lower $\alpha$ is a lazy solution if ever there was one. Certainly, it sounds appealing for clinical psychologists for whom most research of the past decade did not pass a red-face test for replication - sadly, they still won't replicate with more stringent significance. $\endgroup$
    – AdamO
    Feb 29 at 14:48

2 Answers 2

10
$\begingroup$

There is very likely no "simple" function mapping the old sample size to the new one. The way to go about this would be to calculate the effect size detectable at the "old" sample size, then run a power analysis to determine the "new" sample size to detect the same effect size at the smaller alpha level. (Note that this requires the power as an input, are you assuming $\beta=0.8$?) In principle, everything is right there, and one could just calculate and tabulate this.

For instance, here is this for $\beta=0.8$, assuming a two sample t test and testing two-sided (numbers are sample sizes per group):

   sample_old sample_new
1           5   8.609385
2           6  10.349310
3           7  12.074625
4           8  13.791382
5           9  15.503344
6          10  17.211359
7          11  18.916952
8          12  20.620779
9          13  22.323374
10         14  24.025609
11         15  25.725727
12         16  27.425709
13         17  29.125374
14         18  30.825115
15         19  32.523116
16         20  34.219696
17         25  42.710947
18         30  51.196739
19         35  59.682720
20         40  68.166183
21         45  76.648920
22         50  85.138773
23         55  93.614972
24         60 102.095654
25         65 110.568191
26         70 119.066746
27         75 127.537315
28         80 136.028869
29         85 144.502185
30         90 152.983111
31         95 161.463618
32        100 169.943966

R code:

library(pwr)
original_sample_sizes <- c(5:20, seq(25,100,by=5))
pwr <- 0.8
alpha_old <- 0.05
alpha_new <- 0.005

transform_sample_size <- Vectorize(function(sample_size, alpha_old, 
                                             alpha_new, pwr) {
    effect_size <- pwr.t.test(n=sample_size, sig.level=alpha_old, 
                              power=pwr, type="two.sample", 
                              alternative="two.sided")$d
	pwr.t.test(d=effect_size, sig.level=alpha_new, power=pwr, 
     type="two.sample", alternative="two.sided")$n
}, vectorize.args="sample_size")

data.frame(
    sample_old=original_sample_sizes, 
    sample_new=transform_sample_size(original_sample_sizes, alpha_old, 
                                     alpha_new, pwr))
$\endgroup$
10
$\begingroup$

As you wrote ".. there is no "simple" function mapping the old size to the new one .." --this is what I expected. However, following your suggestion and then plotting new against original saple size, one sees for the range n_original 6:100 (which is a reasonable range I am interested in) an "almost" linear relationship.

enter image description here

So, dividing new_n by original_n I get a value of approximately 1.7 throughout the range (1.72 for an original_n of 6 and very slowly decreasing to 1.699 for an original_n of 100). So, for a power of 0.8 (and a reasonable sample size) I can use as a rule-of-thumb the factor 1.7. For power of 0.9 I would get a factor of 1.6. Not an exact solution but a good-enough approximation for a rule of thumb!

R code:

get_new_n <- function(original_n){
    d <- power.t.test(n = original_n, sig.level = 0.025, power = 0.8 , 
                      type ="two.sample", alternative = "one.sided")$delta
    power.t.test(d = d, sig.level = 0.0025, power = 0.8 , 
                 type = "two.sample", alternative = "one.sided")$n
}   
new_n <- sapply(seq(6,100), get_new_n)
plot(seq(6,100), new_n)
new_n/seq(6,100)
$\endgroup$
3
  • 2
    $\begingroup$ Nice! I do wonder whether such linearity holds for more complicated situations, like ANOVA. $\endgroup$ Feb 28 at 16:28
  • 5
    $\begingroup$ It holds for all single-parameter tests, but not for multiparameter tests. The basic reason is the same reason everything is the same in statistics: everything is asymptotically equivalent to a Gaussian location-shift problem. So all one-parameter tests are the same in large samples, but they aren't the same as multiparameter tests. And in small samples discreteness will matter (eg for Fisher's exact test) $\endgroup$ Feb 29 at 6:40
  • 2
    $\begingroup$ Rain on Parade Dept.: The answers so far assume that group sample sizes are equal. $\endgroup$
    – rolando2
    Feb 29 at 19:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.