0
$\begingroup$

I would like to know about a strategy for estimating sample sizes when you're not concerned with statistical significance hypothesis testing (e.g. along the lines in Abandon statistical signficance - McShane et. al.)

e.g. I have a control (C) and a treatment group (T). I would like to test the hypothesis that the treatment group mean is larger than the control group mean and to estimate an effect size. I have some prior information that suggests the difference in means is $\tau = 0.5$. I have a homogenous population to choose from (so that stratifying is not a problem).

I have no intention of doing null hypothesis significance testing so I won't be rejecting/accepting alternative hypotheses. Basing my sample size calculation on type I and type II errors is therefore out.

I do however intend to report a p value for the null hypothesis of the control and treatment being equal, as well as standard errors on the effect size estimates.

My thoughts on estimating and justifying a sample size are as follows:

  1. Choose a sample size $n$ based the standard error ($s.e.$) and the assumed variance ($\sigma^2$). e.g. $$ n = \sigma^2/s.e.^2 $$
  2. The standard error shouldn't be so large as to be meaningless.
  3. The standard error shouldn't be so small as to be impractical.

My question is: how do a expand on points 2 and to some extent 3 above without recourse to statistical significance?

I can make comparisons of the $s.e.$ to things like: 1. measurement uncertainty (i.e. there's no point making it less than measurement uncertainty); 2. to practically significantly differences, (e.g. $s.e. < $ practically significant difference).

Anything else I'm missing?

Thanks in advance.

$\endgroup$
  • 3
    $\begingroup$ I have no intention of doing null hypothesis significance testing [...] I do however intend to report a p value for the null hypothesis of the control and treatment being equal. There are two problems with these statements. (1) The latter, if possible, would be a hypothesis test like any other, so you aren't moving away from NHST with this approach. (2) You cant report a $p$-value for a hypothesis being true. A $p$-value is evidence against a hypothesis. It is the core of NHST. (...) $\endgroup$ – Frans Rodenburg Aug 5 '18 at 12:10
  • 3
    $\begingroup$ (...) The only way to 'test' whether a difference is sufficiently close to zero is by a test for equivalence (two one-sided tests). Moving away from NHST means to report effect sizes, confidence intervals/credibility ranges or reporting predictive accuracy. To be honest, I don't think there is anything intrinsically wrong with using a hypothesis test, as long as you include the effect size and control the false discovery rate. $\endgroup$ – Frans Rodenburg Aug 5 '18 at 12:12
  • $\begingroup$ @FransRodenburg - thanks for your comments 1) I'm talking about the Neyman-Pearson accept/reject framework. 2) I realise that my language was just sloppy. $\endgroup$ – Robert Arbon Aug 6 '18 at 6:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.