I would like to know about a strategy for estimating sample sizes when you're not concerned with statistical significance hypothesis testing (e.g. along the lines in Abandon statistical signficance - McShane et. al.)
e.g. I have a control (C) and a treatment group (T). I would like to test the hypothesis that the treatment group mean is larger than the control group mean and to estimate an effect size. I have some prior information that suggests the difference in means is $\tau = 0.5$. I have a homogenous population to choose from (so that stratifying is not a problem).
I have no intention of doing null hypothesis significance testing so I won't be rejecting/accepting alternative hypotheses. Basing my sample size calculation on type I and type II errors is therefore out.
I do however intend to report a p value for the null hypothesis of the control and treatment being equal, as well as standard errors on the effect size estimates.
My thoughts on estimating and justifying a sample size are as follows:
- Choose a sample size $n$ based the standard error ($s.e.$) and the assumed variance ($\sigma^2$). e.g. $$ n = \sigma^2/s.e.^2 $$
- The standard error shouldn't be so large as to be meaningless.
- The standard error shouldn't be so small as to be impractical.
My question is: how do a expand on points 2 and to some extent 3 above without recourse to statistical significance?
I can make comparisons of the $s.e.$ to things like: 1. measurement uncertainty (i.e. there's no point making it less than measurement uncertainty); 2. to practically significantly differences, (e.g. $s.e. < $ practically significant difference).
Anything else I'm missing?
Thanks in advance.