I would like to calculate a suitable sample size for experiments where multiple testing is an issue and the FDR is used as a measure of significance. I have found a paper (http://bioinformatics.oxfordjournals.org/content/21/23/4263.full) about calculating sample sizes in this situation, and an accompanying R package (manual http://www.stjuderesearch.org/depts/biostats/software/fdrsampsize/fdrsampsize-manual.pdf).

Below is some example code demonstrating an attempted sample size calculation for a correlation analysis:

# this is for ease of demonstration - I actually used 164 correlation coefficients from a previous experiment
> effect.sizes <- c(0.1, 0.2, 0.3, 0.8, 0.8)

> sample.size <- fdr.sampsize (fdr=0.05, ave.pow=0.8, eff.size=effect.sizes, pow.func=power.tcorr)$n 

However, the paper is beyond my statistical understanding and I'm not confident that I am putting in the right numbers.

What I am struggling with is that in a sample size calculation for a single hypothesis, you put in a threshold effect size. For example, for a correlation analysis, you might say 'What sample size do I need to say that I've got a significant correlation with a (threshold) coefficient of 0.8 (with power=0.8 and alpha=0.05)?'

However, with the fdr.sampsize function from the fdrsampsize package there seems to be nowhere to specify a threshold effect size. In the code above, is sample.size the sample size required to say that any of these correlation coefficients that are not zero (i.e. all of them) is significant? If so, this is not what I want - I am happy to just say that the top correlation coefficients (e.g. those above 0.8) are significant.

If this is the case, could I get a suitable sample size by reducing the power parameter to be equal to the fraction of correlation coefficients in the list that are above 0.8? So for the example above, could I use ave.pow=0.2? Or alternatively should I set any coefficients that I don't want to call significant to zero, i.e. effect.sizes <- c(0, 0, 0, 0.8, 0.9)? The demo data for the package seems to use coefficients with a value of zero to represent cases where the null hypothesis is true, but I don't know if this is just for ease of demonstration.

The demo data also uses a power of 50%, and I don't know where that has come from, as 80% is standard for stats tests.

Also, in the paper it talks about calculating the effect.sizes parameter from background data, but it sounds very complicated, so maybe I have taken too simple an approach in just using the correlation coefficients from example data.

So basically my question is: what are the appropriate numbers to use for the ave.pow and eff.size parameters?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.