Can anyone help with justifying (or rubbishing) a couple of aspects of a sample size calculation for a grant proposal. A pilot study gave significant results for the correlation of a continuous (non-normal) outcome with an ordinal variable, using Kendall's Tau. The first stage of a new study is reproducing the results with a new set of specimens. I calculated the required sample size using resampling of the pilot results and using alpha = 0.1% and Power = 95%. I have two questions:
I thought it prudent to use those fairly stringent values of alpha and beta (rather than the conventional 5% and 20% values) given that it is a reproducibility study. However, I was asked by a proper statistician (I am not!) to justify those choices. To me it is common sense to use lower values for a robust reproducibility study, for a whole range of reasons, and there is obviously not a "correct" choice of alpha and beta. However, is there a good reference that supports the use of lower values when conducting a reproducibility study? I do not have room in the proposal for an essay on the subject!
I was also asked to justify the use of resampling rather than "more traditional methods". My method involved taking 10^4 samples of size n (with replacement) from each independent level of the pilot data, applying Kendall, and counting the p < 0.001 hits to calculate the power for that sample size. This obviously assumes that the effect size in the pilot is representative of the population, but that does seem to a sensible use of pilot data rather than making arbitrary assumptions about effect size. I have found various oblique references in the literature to this approach, but not a killer citation, which is what I am after, if anyone is aware of one? Maybe there is no such thing because that approach is flawed?