Sample size calculation for Kendall's Tau in reproducibility study

Question

Can anyone help with justifying (or rubbishing) a couple of aspects of a sample size calculation for a grant proposal. A pilot study gave significant results for the correlation of a continuous (non-normal) outcome with an ordinal variable, using Kendall's Tau. The first stage of a new study is reproducing the results with a new set of specimens. I calculated the required sample size using resampling of the pilot results and using alpha = 0.1% and Power = 95%. I have two questions:

1. I thought it prudent to use those fairly stringent values of alpha and beta (rather than the conventional 5% and 20% values) given that it is a reproducibility study. However, I was asked by a proper statistician (I am not!) to justify those choices. To me it is common sense to use lower values for a robust reproducibility study, for a whole range of reasons, and there is obviously not a "correct" choice of alpha and beta. However, is there a good reference that supports the use of lower values when conducting a reproducibility study? I do not have room in the proposal for an essay on the subject!

2. I was also asked to justify the use of resampling rather than "more traditional methods". My method involved taking 10^4 samples of size n (with replacement) from each independent level of the pilot data, applying Kendall, and counting the p < 0.001 hits to calculate the power for that sample size. This obviously assumes that the effect size in the pilot is representative of the population, but that does seem to a sensible use of pilot data rather than making arbitrary assumptions about effect size. I have found various oblique references in the literature to this approach, but not a killer citation, which is what I am after, if anyone is aware of one? Maybe there is no such thing because that approach is flawed?

Answer 1

1. I am not aware of any convention to use more stringent alpha levels (and shoot for higher beta) in reproducibility studies, but this may certainly be specific to the field.

   The drawback of a lower alpha and higher beta is, of course, that you will need more data, i.e., a larger sample size and more resources to conduct your study. A statistical reviewer might run his own calculations and argue that the resources you are applying for can be reduced by, say, 75% by assuming "standard" alpha and beta values.

   So I would suggest that you need to argue why the more stringent values are useful. Is there a danger for this to become an underpowered study?

   And of course, in a Bayesian sense, the fact that you are reproducing a previously reported effect means that our priors are already shifted in favor for there being an effect. Which would argue for more lenient alpha and beta levels, not more stringent ones.

2. I personally believe that judicious resampling is a much better approach to power analysis than using canned formulas or online sample size calculators. It allows you to tailor your power analysis to your specific study and to the specific data you have. I would say that using a "traditional" canned formula should require justification, not the other way around.

   Then again, I likely won't be reviewing your grant application.

   I would recommend that you keep your resampling approach and justify it in a short sentence as per my paragraph above. For bonus points, look whether there are any established power calculations for your specific problem, taking the effect size from your pilot data - these calculations will likely enough yield a roughly similar result to your resampling. If nothing else, reporting both should favorably impress any reviewers with your thoroughness.

Finally, do not be wedded to the effect size you observe in your pilot data. (It's already better than taking one reported in the literature, which would certainly suffer from publication bias.) Rather, use an effect size that is clinically relevant - whether that is larger or smaller than what you see in your pilot data. The best guidance for choosing an effect size I have come across is to use an effect size that one would be sorry to miss because of underpowering the study.