Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm about to do a study about the merits of one stimulus compared to another with a within-subject design. I have a permutation scheme that is designed to reduce order effects of some parts of the study (task type order, stimulus order, task set order). The permutation scheme dictates that the sample size be divisible by 8.

To determine the sample size I'd have to either take a wild guess (being a good tradition in my field) or calculate the sample size for my desired power. The problem is now that I don't have the slightest clue how large an effect size I'm going to observe (also a good tradition in my field). That means power calculation is a bit difficult. Taking a wild guess, on the other hand, might also be bad because I can either come out with too low a sample size or pay too much money to my participants and spent too much time in the lab.

Is it okay to state upfront that I add participants in batches of 8 persons until I leave a corridor of two p-values? E.g. 0,05 < p < 0,30? Or how else, would you recommend, should I proceed?

share|improve this question
It sounds like you might be looking for something in the realm of a sequential design/analysis. – cardinal Oct 18 '11 at 19:23
Thanks everyone for your insight. I it were possible to accept multiple answers I'd do it :-( – xmjx Oct 19 '11 at 22:31
up vote 8 down vote accepted

First, to answer your question directly: no, you cannot just keep going until you get a significant p-value. The design you propose has a type I error rate above 5%. However the underlying idea is correct, except you have to adjust the cutoffs. In fact, as @cardinal mentioned in the comments, there is an entire field of research for your question: these are called sequential, or group-sequential, or more generally, adaptive designs (they are not the same things, but all along the lines of your idea).

Here is a reference that demonstrates some of the basic ideas: C. Mehta, P. Gao, D. L. Bhatt, R. A. Harrington, S. Skerjanec, J. H. Ware Optimizing Trial Design: Sequential, Adaptive, and Enrichment Strategies Circulation. 2009; 119: 597-605

share|improve this answer
Okay, that sounds interesting. I read about adaptive designs before but only on some slides so the paper reference you gave looks really useful. – xmjx Oct 19 '11 at 22:22

Have you considered looking at power over a range of effect sizes? For example, I frequently calculate power as a curve, and end up with a myriad of potential scenarios baked into the graph, wherein I can then make a sample size decision. For example, I might calculate the needed sample size for effect measures ranging from very close to null to slightly higher than my wildest, this-will-sail-through-peer-review dreams.

I might also plot other scenarios, depending on how much I don't know about the data. For example, below is a plot that calculates power, not sample size, but has a similar concept to it. I know very little about the data, so I assumed a 10% event rate for a survival analysis, and then calculated the study's power (sample size was fixed) over a number of conditions:

Power Curves

One might also even be able to vary in this case the number of events, which would leave you with either multiple plots, or a "Power Surface". That seems to be a much quicker way to get a handle on at least where you should be looking for sample size, rather than modifying sample size on the fly. Or at least give you a threshold where you can stop adding people. For example, if your calculations tell you 1,000 people will let you see an effect of something very small - for example, a hazard ratio of 1.01 or the like - you know that if you hit that, you can stop trying to add people, because its not a power problem, but a "There's nothing there" problem.

share|improve this answer
That's a really great idea. While I side with @Aniko's answer because it addresses my question directly, your suggestion is definitely more useful in the short term. I think it will take me some time to wrap my mind around sequential strategies. – xmjx Oct 19 '11 at 22:25

When doing power calculations, the question I usually ask (in my field, which also has these traditions) tends to be "How big would an effect have to be for people to care?". If your method is "significantly" better with a 0.1% improvement, will anyone care? How about a 0.01% improvement?

share|improve this answer
I'll use that train of thought to get a starting point for @EpiGrad's power plot. – xmjx Oct 19 '11 at 22:28

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.