Is there any way to do a power-based sample size calculation if you don't know the minimal (scientifically) important difference?

Question

I am in the life sciences field and I don't have much experience with power-based sample size calculations. I am doing a Western blot experiment and my research question is whether protein X influences the levels of protein Y during development. In this experiment, I have two groups of mice (that differ based on their genotype), Genotype A and Genotype B. These groups are further subdivided by age, so in total I have four groups of mice:

1. Genotype A, 10 day old mice
2. Genotype A, 20 day old mice
3. Genotype B, 10 day old mice
4. Genotype B, 20 day old mice

My outcome measure is the levels of protein Y in the brains of these mice. In this study, I am interested in seeing a difference between any pair of the 4 groups. I am using the two-way ANOVA test to analyse my data since I am interested in seeing how age and genotype in combination affects the levels of protein Y.

I want to do a sample size calculation so that my study has enough power to detect a meaningful difference in protein Y levels between these groups of mice. I am not that experienced with power-based sample size calculations but I know that you need to know the following things: null hypothesis, alternative hypothesis, statistical test (two-way ANOVA), desired type 1 error rate, desired power and minimal scientifically important difference.

My desired type 1 error rate is 0.05 and my desired power is 80%. I know my null & alternative hypotheses and I'm using a two-way ANOVA followed by Sidak's multiple comparisons post hoc test. However, the only thing that I don't know is the minimal scientifically important difference (Hm). I know that you can look at previous published studies to get an idea of Hm but there are no previously published studies similar to what I am doing. Further, this is the first time I am doing these experiments so I am not sure what Hm is myself (because I have read that you can do a pilot study to find Hm if it's not something you can find out from the literature).

I know that you can do a power-based sample size calculation using the G*Power software. However, in this software, you need to know what the minimal scientifically important difference is to calculate sample size that will give you the desired power. I know that if you don't know what the meaningful difference is, you can do a sensitivity power analysis - where you calculate the effect size you can detect with a certain number of mice. However, at this point of the experiment, this information is not useful.

I was wondering is there any way at all to do a power-based sample size calculation if you don't know what the minimal scientifically important difference is? E.g. what sample size would I need to have a power of 80%?

Any advice is appreciated.

Answer 1

As an aside type I error rate $\alpha$ is neither a rate nor an error. I prefer to call it type I assertion probability.

The answer to your question is no. But you could use slightly different wording for the parameter that needs to go into the power calculation (besides $\alpha, n, \sigma$). Call it the effect you would be embarrassed to miss. This may harmonize with the minimal biologic effect in many cases.

For many problems, possibly including yours, power is a less useful concept than estimating the likely information yield of the experiment. A simple way to think of information yield is the margin of error in estimating a key quantity of interest. This quantity may be a difference in means between two groups, and correspond to half the width of, say, a 0.95 confidence interval. This approach is usually called planning for precision, and I delve into this in BBR. Instead of a "difference not to miss" you have to specify an acceptable margin of error. This is not necessarily a trivial task but is a bit easier than agreeing on a biologic effect not to miss. And it is more useful in my view, because if you run the experiment and get p=0.12 you have no idea what to conclude, but the compatibility (confidence) interval has meaning whether you accept or reject $H_0$.