How does reducing my alpha-threshold to 0.005 affect my required sample size?

Question

Among statisticians there is an ongoing debate whether the long-standing "standard" threshold for declaring the outcome of a statistical test as "significant" (a.k.a. alpha-level) should be reset from the current default value of p<0.05 to p<0.005. This suggestion is floating around for a while but has ben specifically expressed in a paper published some years ago: Benjamin, et al. "Redefine statistical significance." Nature human behaviour 2.1 (2018): 6-10. https://doi.org/10.1038/s41562-017-0189-z

If I want to do this without losing too much power, this will require that I adjust my sample size accordingly. My question is: For the simplest case of a two-sample t-test and a current sample size X (that is deemed sufficient to detect an effect with alpha=0.05), is there a way to say in general by how much I have to increase my sample size if I switch to alpha=0.005? Specifically --is there a way to say that, that does not require making a power-analysis for a specific case?

Answer 1

There is very likely no "simple" function mapping the old sample size to the new one. The way to go about this would be to calculate the effect size detectable at the "old" sample size, then run a power analysis to determine the "new" sample size to detect the same effect size at the smaller alpha level. (Note that this requires the power as an input, are you assuming $\beta=0.8$?) In principle, everything is right there, and one could just calculate and tabulate this.

For instance, here is this for $\beta=0.8$, assuming a two sample t test and testing two-sided (numbers are sample sizes per group):

   sample_old sample_new
1           5   8.609385
2           6  10.349310
3           7  12.074625
4           8  13.791382
5           9  15.503344
6          10  17.211359
7          11  18.916952
8          12  20.620779
9          13  22.323374
10         14  24.025609
11         15  25.725727
12         16  27.425709
13         17  29.125374
14         18  30.825115
15         19  32.523116
16         20  34.219696
17         25  42.710947
18         30  51.196739
19         35  59.682720
20         40  68.166183
21         45  76.648920
22         50  85.138773
23         55  93.614972
24         60 102.095654
25         65 110.568191
26         70 119.066746
27         75 127.537315
28         80 136.028869
29         85 144.502185
30         90 152.983111
31         95 161.463618
32        100 169.943966

R code:

library(pwr)
original_sample_sizes <- c(5:20, seq(25,100,by=5))
pwr <- 0.8
alpha_old <- 0.05
alpha_new <- 0.005

transform_sample_size <- Vectorize(function(sample_size, alpha_old, 
                                             alpha_new, pwr) {
    effect_size <- pwr.t.test(n=sample_size, sig.level=alpha_old, 
                              power=pwr, type="two.sample", 
                              alternative="two.sided")$d
	pwr.t.test(d=effect_size, sig.level=alpha_new, power=pwr, 
     type="two.sample", alternative="two.sided")$n
}, vectorize.args="sample_size")

data.frame(
    sample_old=original_sample_sizes, 
    sample_new=transform_sample_size(original_sample_sizes, alpha_old, 
                                     alpha_new, pwr))

Answer 2

As you wrote ".. there is no "simple" function mapping the old size to the new one .." --this is what I expected. However, following your suggestion and then plotting new against original saple size, one sees for the range n_original 6:100 (which is a reasonable range I am interested in) an "almost" linear relationship.

So, dividing new_n by original_n I get a value of approximately 1.7 throughout the range (1.72 for an original_n of 6 and very slowly decreasing to 1.699 for an original_n of 100). So, for a power of 0.8 (and a reasonable sample size) I can use as a rule-of-thumb the factor 1.7. For power of 0.9 I would get a factor of 1.6. Not an exact solution but a good-enough approximation for a rule of thumb!

R code:

get_new_n <- function(original_n){
    d <- power.t.test(n = original_n, sig.level = 0.025, power = 0.8 , 
                      type ="two.sample", alternative = "one.sided")$delta
    power.t.test(d = d, sig.level = 0.0025, power = 0.8 , 
                 type = "two.sample", alternative = "one.sided")$n
}   
new_n <- sapply(seq(6,100), get_new_n)
plot(seq(6,100), new_n)
new_n/seq(6,100)