7
$\begingroup$

I'm doing a cross-sectional descriptive study with an estimated sample size being 140. Will it be statistically justifiable if I managed to collect a sample of 200? Could there be any side effect that may not be scientifically acceptable?

$\endgroup$
4
  • 1
    $\begingroup$ No, what would be the scientific unacceptable side effect of more data? The only real down side is that its gonna cost money $\endgroup$
    – Repmat
    Commented May 3, 2016 at 6:52
  • $\begingroup$ (This doesn't seem to me that it is so unclear that it needs to be closed.) $\endgroup$ Commented May 3, 2016 at 15:39
  • 1
    $\begingroup$ It is invalid if you notice your results aren't quite significant, so you keep gathering data to get significance. If that isn't what you're doing, I don't see the problem here. $\endgroup$ Commented May 3, 2016 at 15:40
  • 1
    $\begingroup$ This question of mine, particularly the answer by the author of the paper where I got confused, is worth a read. $\endgroup$
    – Dave
    Commented Apr 9 at 13:03

2 Answers 2

11
$\begingroup$

I can think of at least three possible issues with exceeding a planned sample size, but you will have to judge for yourself, which of these apply in your case.

  1. Perhaps most importantly, there might be ethical issues. E.g. if more samples means more animals or humans being involved in an experiment, then this might be problematic, if an ethics committee approved a much smaller sample size.
  2. A key concern from a statistical perspective is that if this is done based on what the collected data looks like, then there is a problem with keeping type I error rate etc. and approaches like group-sequential methods or adaptive designs might be needed. That should ideally be decided up-front. However, that is not a concern, if your approach is "Without looking at the data we are getting we will collect at least 140 samples and more if we can, then we will go and analyse what we have." I would be tempted to clearly say that in my research protocol though.
  3. It might not be entirely necessary and be a waste of money/resources/effort to collect more samples than planned. If there were a very large discrepancy, the result might mean that you demonstrate very clearly that there is some effect/difference (whatever you are looking for) that is so small that it is irrelevant. However, 200 instead of 140 does not sound like a massive difference in terms of power for the effect size you may have powered for (this would not turn 80% power into 99.9%) and is not going to get you an enormous power for a much smaller effect size.
$\endgroup$
8
$\begingroup$

To add to Björn's answer (+1), exact tests involving discrete random variables can lead to fluctuations in statistical power as sample size increases. So if you plan to use such a test, it might be a good idea to plot power against sample size to make sure nothing unexpected will happen if you decide to use a larger sample size.

Below is an example of power plotted against sample size for a McNemar test, showing that adding a couple of observations after collecting 140 observations can unintuitively decrease power, despite its overall increasing trend. (All assumptions are included in the image). In this example, reaching 200 observations obviously does not lead to a loss of power relative to 140 observations, but adding a couple of observations does. So it really depends on the kind of test you'll be using (if any), what are your assumptions, and on how you intend to increase your sample size:

The plot shows how power behaves relative to sample size for a two-tailed McNemar test, with a proportion of discordant pairs of 0.35, an alpha of 0.01, and an odds ratio of 3.1. Sample size is plotted on the x axis, and power on the y axis. For a sample size of 140, the power equals 0.88, but drops to 0.86 if we increase the sample size to 141 or 142, down to 0.84 with a sample size of 143, 144, or 145. After that, power increases to 0.89 for sample sizes of 146, 147, and 148, but drops again to 0.87 for sample sizes of 149, 150, and 151. When reaching a sample size of 152 and beyond, power does not drop below 0.88 again, but continues to have the "saw-tooth" behavior we observe, despite an overall increasing trend.

(The graph has been generated with G*Power.)

If you don't have a lot of control over the precise number of observations you collect (e.g. if you're conducting a study where you could get accidentally a few more participants than expected), but don't want power to drop below a certain, then this kind of plot can help you plan a minimum number of observations to collect. For instance, in the plot above, if you don't want power to drop below 0.88, then you have to collect at least 152 observations. 140 observations would give you a power of 0.88 indeed, but power would drop to 0.84 with just 3 "accidental" additional observations.

On the other hand, if you have a very precise control over your sample size, a different strategy can be more relevant depending on the circumstances on your study. For instance, if you decide to collect exactly 200 observations, notice that 186 observations give you a bit more power than 200 observations. That's something to consider relative to cost or ethical issues, as described in Björn's answer.

For a reference discussing all these issues, you can have a look at: Chernick, M. R., & Liu, C. Y. (2002). The Saw-Toothed Behavior of Power Versus Sample Size and Software Solutions. The American Statistician, 56(2), 149–155. https://doi.org/10.1198/000313002317572835.


As a side note, as you were asking about possible side effects, an indirect and possibly underestimated cost is an increase of survey fatigue in your population of interest, if you're conducting surveys on human beings.

On the long run, survey fatigue can increase non-response rates in future surveys, as people get tired of being surveyed again and again. So that is another practical (and possibly ethical) matter to think of, if you're considering increasing your sample size.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.