Is dynamically increasing the sample size okay, if stated a priori? I'm about to do a study about the merits of one stimulus compared to another with a within-subject design. I have a permutation scheme that is designed to reduce order effects of some parts of the study (task type order, stimulus order, task set order). The permutation scheme dictates that the sample size be divisible by 8.
To determine the sample size I'd have to either take a wild guess (being a good tradition in my field) or calculate the sample size for my desired power. The problem is now that I don't have the slightest clue how large an effect size I'm going to observe (also a good tradition in my field). That means power calculation is a bit difficult. Taking a wild guess, on the other hand, might also be bad because I can either come out with too low a sample size or pay too much money to my participants and spent too much time in the lab.
Is it okay to state upfront that I add participants in batches of 8 persons until I leave a corridor of two p-values? E.g. 0,05 < p < 0,30? Or how else, would you recommend, should I proceed?
 A: First, to answer your question directly: no, you cannot just keep going until you get a significant p-value. The design you propose has a type I error rate above 5%. However the underlying idea is correct, except you have to adjust the cutoffs. In fact, as @cardinal mentioned in the comments, there is an entire field of research for your question: these are called sequential, or group-sequential, or more generally, adaptive designs (they are not the same things, but all along the lines of your idea).
Here is a reference that demonstrates some of the basic ideas: C. Mehta, P. Gao, D. L. Bhatt, R. A. Harrington, S. Skerjanec, J. H. Ware Optimizing Trial Design: Sequential, Adaptive, and Enrichment Strategies Circulation. 2009; 119: 597-605
A: Have you considered looking at power over a range of effect sizes? For example, I frequently calculate power as a curve, and end up with a myriad of potential scenarios baked into the graph, wherein I can then make a sample size decision. For example, I might calculate the needed sample size for effect measures ranging from very close to null to slightly higher than my wildest, this-will-sail-through-peer-review dreams.
I might also plot other scenarios, depending on how much I don't know about the data. For example, below is a plot that calculates power, not sample size, but has a similar concept to it. I know very little about the data, so I assumed a 10% event rate for a survival analysis, and then calculated the study's power (sample size was fixed) over a number of conditions:

One might also even be able to vary in this case the number of events, which would leave you with either multiple plots, or a "Power Surface". That seems to be a much quicker way to get a handle on at least where you should be looking for sample size, rather than modifying sample size on the fly. Or at least give you a threshold where you can stop adding people. For example, if your calculations tell you 1,000 people will let you see an effect of something very small - for example, a hazard ratio of 1.01 or the like - you know that if you hit that, you can stop trying to add people, because its not a power problem, but a "There's nothing there" problem.
A: When doing power calculations, the question I usually ask (in my field, which also has these traditions) tends to be "How big would an effect have to be for people to care?". If your method is "significantly" better with a 0.1% improvement, will anyone care? How about a 0.01% improvement?
