3
$\begingroup$

We are trying to figure out sample size to our research. We are comparing to groups. Group 1 will get treatment and group 2 don't get treatment. From previous studies we know that there was recurrence in the treatment at group 1 at 1% per cases and group 2 6% per cases.

I run a test for sample sizes with comparing proportions (Inference for Proportions: Comparing Two Independent Samples, https://www.stat.ubc.ca/~rollin/stats/ssize/b2.html). I used proportions from previous studies, group 1 0.01 and group 2 0.06. It calculates that we need to get 211 patients per group. If we collect 211 patients it will take too much time and resources.

For us it would be realistic to collect about 40-60 patients per group. Is there any other possibilities to do this? Can I use the effect size? And how can I calculate it?

$\endgroup$
3
  • 2
    $\begingroup$ The real question is why you ever performed a sample size calculation when you knew the realistic sample size was limited. $\endgroup$
    – AdamO
    Commented Jan 21 at 6:41
  • 2
    $\begingroup$ Broadly speaking there are two elements that determine sample size for a given power: the size of the difference, and its variability (their ratio is the effect size). Your outcomes being proportions their variance is fixed by their absolute value, so your only option is to power for a larger difference (or accept lower power). $\endgroup$
    – PBulls
    Commented Jan 21 at 11:45
  • 1
    $\begingroup$ @AdamO I don't agree. The reason to do a sample size calculation is to figure out if they should spend time and money doing this study and trying to do a hypothesis test. $\endgroup$
    – Peter Flom
    Commented Jan 21 at 12:04

1 Answer 1

2
$\begingroup$

Short answer: No. The power analysis is telling you not to spend time and effort on this. This is, at first, very disappointing. But one reason to do a power analysis is that you want to figure out if it's worth doing the research.

Longer answer: Part 1 - can you increase power? Well, maybe. @PBulls was correct in the comments about the two ways to increase power and that, for your test, variability is fixed. But maybe you don't have to do a test of proportions. Maybe there are useful covariates and you could do some kind of regression. Covariates can be added to a model for various reasons, one is to reduce noise.

Or maybe you could change your design. Maybe you can do some kind of repeated measures, where both groups get control and treatment, but in different orders (whether this is feasible depends on what they are being treated for and what the surgery is).

Or maybe you could do a survival analysis, with "time to recurrence" as the DV.

Part II - You could abandon the idea of hypothesis testing and, instead, simply treat this as a sort of pilot study where you estimate the effect size and its standard error. If you find a big effect size, you say "more research is needed" and (maybe) "please give us tons of money to investigate it."

$\endgroup$
6
  • 2
    $\begingroup$ I don’t think that doing a pilot study allows you to adequately estimate an effect size. To the point of how to decrease the needed sample size, there are two solutions with the highest payoff: do a long-term longitudinal study or pick a response variable that is nearly continuous and has high resolution and high test-retest reliability. This greatly increases the effective sample size. Binary endpoints have minimum statistical information and power. See BBR for more. $\endgroup$ Commented Jan 21 at 13:51
  • 2
    $\begingroup$ In my experience, pilot studies are often used to get some crude estimate of an effect size and justify more extensive work. Of course, that's not the only reason to do them, but it is one. Even an estimate with a very large standard error can be a bit useful. $\endgroup$
    – Peter Flom
    Commented Jan 21 at 14:28
  • $\begingroup$ Thank you for answers. Like @Peter Flom said it would be frustrating to abandon the research idea if statistical model says there's not enough effect to do that. I'm wondering if this is common for researches that they have a good research plan (at least for their own opinion) and they need to abandon it because the sample size would be to big. $\endgroup$
    – user405350
    Commented Jan 21 at 19:38
  • $\begingroup$ If I change the study design and do a survival analysis with "time to recurrence" as the DV, do we need to use sample size calculation also to this study design? $\endgroup$
    – user405350
    Commented Jan 21 at 19:44
  • 1
    $\begingroup$ @user405350 yes, it's very common, and ultimately, it's a good thing. There's no point in running studies with sample sizes that have no hope of providing reliable evidence of the effect of interest. In my field it used to be common to use too small samples and now the whole field is in crisis due to a large proportion of results being unreliable. $\endgroup$
    – Sointu
    Commented Jan 23 at 8:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.