13
$\begingroup$

Honestly, we didn't expect this - after all we did everything by the playbook.

We planned a study on some psychological issue, we entered questionnaires into Qualtrics, and with a help of G*Power we did power analysis to get the minimum sample size and finally we posted a link to the study on the internet. And then the sample size exploded. After a few days we checked how many observations we had managed to collect and it turned out that the number had quadrupled (so we hastily stopped collecting the data). The power analysis indicated N = 500 and we got N = 2000 (2000-something, of course). Happy? No, we are very far from being happy.

Q: So now we have a problem of what to do with an oversized sample (keeping in mind overpowered studies).

The ideas are:

  1. cut off at 500th observation -- and discard the rest (sounds like a waste of data)
  2. cut off at 500th observation and use the rest as a replication study (sounds sneeky, we actually didn't do a replication study, the data comes from the original one)
  3. to sample a sample - sample without replacement from our big data (with N = 2000) a sample consisting of N = 500 observations (and discard the rest).
  4. ... any other thoughts, guys? Have you ever been in that situation? What did you do?
Improve this question
$\endgroup$
15
  • 29
    $\begingroup$ Why is it a problem to have more data than you planned on?? This is the statistical analog of budgeting 2000 dollars to purchase equipment and managing to buy it on sale for 500 dollars. Unhappy, really? $\endgroup$
    – whuber ♦
    Commented Aug 18 at 15:00
  • 6
    $\begingroup$ @Dave It might not be--but there are no such nuances indicated in how the question currently is formulated. Note, too, that Kurschke himself admits that perspective is "unconventional." $\endgroup$
    – whuber ♦
    Commented Aug 18 at 15:07
  • 7
    $\begingroup$ "We are worried about how we are going to explain to a reviewer that in the pre-registration we said N = 500 and got N = 2,000 :) ". Simply tell that you got more observations than expected. Why would a reviewer think it is a problem? I'd be much more worried about the study design, to be honest. $\endgroup$
    – J-J-J
    Commented Aug 18 at 15:58
  • 7
    $\begingroup$ No, not because of the large sample size, but because it's a convenience sample. You should be more concerned by that. // FYI, in this paper, Daniël Lakens provides an example on how to explain that you collected more observations than what was planned in pre-registration: online.ucpress.edu/collabra/article/10/1/117094/200749/… (see the paragraph starting with "Alternatively, a researcher might describe the following deviation from their preregistration if they collected more data than planned") $\endgroup$
    – J-J-J
    Commented Aug 18 at 16:17
  • 3
    $\begingroup$ You write "with G*Power we did power analysis to get the minimum sample size". I'd guess that this analysis was based on the assumption that you have a random sample of the population. But what you describe strongly contradicts that. The number of samples is a red herring. $\endgroup$
    – user2705196
    Commented Aug 19 at 17:49

5 Answers 5

Reset to default
25
$\begingroup$

More data all else being equal means a better answer to a problem. Issues with study design/validity e.g. due to respondents on the internet not being representative of the population of interest etc. may of course not get resolved with more data. However, I cannot think of a situation where they would get worse by having more data.

Some real/perceived problems I can see are:

  • ethics questions (especially if you applied an unproven intervention to by far more people than your ethics committee approval covered - doesn't seem to apply),
  • regulatory/process questions (e.g. if you did this for getting a drug approved, regulatory authorities might have questions about how you didn't have control over your experiment and deviated from your plan - doesn't seem to apply), and
  • concerns about manipulation (in the sense that someone might think you peeked at the data after 500 respondents, didn't like the answer, let it keep going and then stopped when by chance you finally got something you liked - I'll assume this doesn't apply).

From a purely scientific question I'd be tempted to report in a publication that while only a sample size of 500 was needed per the power calculation, respondents accrued so fast that you exceeded that number substantially, and report results based on all the data (perhaps double-check your main finding isn't different when only done on the first 500, in case a reviewer asks you to check that).

Improve this answer
$\endgroup$
3
  • $\begingroup$ I can only say there was no data-peek-ation :) This answer got me thinking that if we had peeked at the data at N = 50 and had been not happy with results we would have been eligible to continue collecting, just because we pre-registered a sample size larger than 50. $\endgroup$
    – Lil'Lobster
    Commented Aug 18 at 16:02
  • 2
    $\begingroup$ To avoid that kind of concern in clinical trials, you would even keep the people doing the analysis (the trial sponsor) blinded to group assignments. If there is a good reason to look, then this is often done by an independent data center giving the analyses to a independent data monitoring committee that decides according to pre-specified rules. These rules might concern when to stop due to side effects, due to a lack of a realistic chance to get a positive trial result or because the answer is already clear (e.g. based on using group-sequential methods). $\endgroup$
    – Björn
    Commented Aug 19 at 5:46
  • 3
    $\begingroup$ @Lil'Lobster The peeking is problematic if you would do something about it, not only if you did do something about it. You could imagine peeking throughout the study and planning to stop as soon as you find the answer you want. Even if you only find the answer you want at the very end once you reach the specified N and do not terminate early, this is a problematic design that is unfairly biased to report the answer you want. $\endgroup$
    – Nuclear Hoagie
    Commented Aug 19 at 13:36
14
$\begingroup$

The key issue here is the misconception regarding "overpowered studies". In almost all circumstances it is not possible for a study to have too much power. 100% power in a statistical study is perfect! The exceptions relate only to wasted expenses of having more data than might be needed and the three items mentioned by Björn in his answer.

There are questions already on this site that provide useful insights. For example, this: Are large data sets inappropriate for hypothesis testing? and here: When is a sample size too large?

My impression is that the idea of an "overpowered study" comes from three things. First is the fact that a very large sample will often give 'significant' results even when the true effect size is trivial. That is a problem only where the significance is reported and the observed effect size is not. That used to happen often with publications relating to psychology and other 'social sciences'. It is a problem that should never have occurred, and should have been eliminated by now. Always show the observed effect size and discuss its importance in the context of the study.

The second factor that contributed to the idea of "overpowered" studies is the fact that in any indefinitely extended sampling procedure the result will at some stage(s) be found 'significant' for any value of $\alpha$ even where the true effect size is zero. If the sampling is stopped when the result is 'significant' then a 'significant' result is certain to result. That is a fact that is sometimes used to discredit the hypothesis testing approach as well as a warning against the "overpowered study". However, simulations show that the vast majority of the sample sizes needed to approach 100% 'significant' results with unrestricted sequential sampling when the true effect size is zero are vastly larger than the sample sizes that are feasible in the real world. There are no useful lessons for study design that come from that.

The third reason that the idea of "overpowered studies" exist is the related idea occasionally pushed by some Bayesians that frequentist hypothesis testing becomes biased against the null hypothesis when the sample size is large. That is a false notion. It is discussed here: Why does frequentist hypothesis testing become biased towards rejecting the null hypothesis with sufficiently large samples? but note that the accepted answer is wrong! See my own answer and the comments for a discussion of why that is the case.

Improve this answer
$\endgroup$
9
  • 1
    $\begingroup$ (+1) but note that the accepted answer is wrong: since answers can be edited and the OP can change what answer to accept, it may be worth adding a link to the current answer as it is phrased now (is that even possible?). Anyway, yes, I also find biased against the null hypothesis to be a very misleading phrasing to an already complex topic. $\endgroup$
    – dariober
    Commented Aug 19 at 7:50
  • 4
    $\begingroup$ +1. I have seen real applications that require underpowered studies. Without naming names (or contexts), they concern corporations that must meet numerical government standards based on hypothesis testing and those corporations are hoping not to reject the null hypothesis... . $\endgroup$
    – whuber ♦
    Commented Aug 19 at 13:16
  • 4
    $\begingroup$ @NuclearHoagie A hypothesis test will not have any bias towards the null for a sample with its size fixed in advance in the normal manner no matter how large the sample size is. For the "indefinitely extended sampling" situation it is assumed that the sampling is stopped once a 'significant' result is observed. It is that optional stopping when a significant result is observed that does the trick of guaranteeing a significant outcome, not the large sample size. I guess I did not spend enough words to make that distinction clear. $\endgroup$
    – Michael Lew
    Commented Aug 19 at 21:16
  • 3
    $\begingroup$ @whuber Yes, that's a good exception to my "almost all". It's also an example of where hypothesis test results should not be used in place of effect sizes! $\endgroup$
    – Michael Lew
    Commented Aug 19 at 21:23
  • 2
    $\begingroup$ @MichaelLew Sorry, I think I wasn't clear in my comment above. I don't suggest you should edit someone else's answer. I meant: if the author of that answer edits it or the OP changes the accepted answer, then your statement "the accepted answer is wrong" gets out of sync. (Anyway, no big deal really). $\endgroup$
    – dariober
    Commented Aug 20 at 8:19
13
$\begingroup$

To add to Björn's answer (+1): In comments, you say you're worried about explaining to a reviewer that you collected more observations than expected. But if you ditch observations, you would also have to explain it to the reviewer, and I suspect it wouldn't look very good.

So the real question here is if it's a good or bad thing to have more observations than planned. I don't see how being able to detect smaller effect sizes and having better precision of estimates would be a problem.

In addition, there would be an ethical issue if you ditched the additional observations you got: you asked people to take time to answer your survey, and ultimately you wouldn't use the data they shared with you. This would be a waste of their time, and this might also jeopardize participation of people in future surveys ("Why would I participate if my answers risk to be ignored?").

What to do when you get much more observations than you expected is to understand why you got much more data (e.g. does it hint to some sort of fraud? a software bug? etc.). If you don't identify an issue with the data collection process, then by all means use the data, and simply explain that you got more observations than expected. Daniël Lakens provides an example in his paper When and How to Deviate From a Preregistration:

[...] a researcher might describe the following deviation from their preregistration if they collected more data than planned:

  1. In our preregistration we expected to collect data from 500 participants based on past experiences recruiting participants through social media. Unexpectedly, our request to collect data went viral, and in the end 5124 participants completed our survey. We only analyzed the data once and included all participants who completed the survey before we started the data analysis.

  2. We performed the analysis as originally planned but had much higher power to detect the effect of interest. This deviation did not reduce the severity of the test but increased it.

References

Daniël Lakens; When and How to Deviate From a Preregistration. Collabra: Psychology 16 January 2024; 10 (1): 117094. doi: https://doi.org/10.1525/collabra.117094

Improve this answer
$\endgroup$
2
  • 3
    $\begingroup$ I like and dislike this paper. On one hand, it offers solutions for when you can amend a pre-registration. On the other hand, I'm more in the camp of just document everything (in either the pre-registration or the manuscript later). This paper seems to make it out that it should be conditional, and echoes what others say about being extremely rigid about the pre-reg not changing. I think that is very limiting and forces people to omit things that "dont fit the bill" in fear of looking like they are engaging in QRPs. But that is more a rant and this answer is still good (+1) $\endgroup$
    – Shawn Hemelstrand
    Commented Aug 19 at 2:34
  • 3
    $\begingroup$ +1 for the "understand why you got more data than expected". Getting a sample several times larger than you expected sort of suggests a gap in understanding of the population being sampled, the sampling method, or both. $\endgroup$
    – Nuclear Hoagie
    Commented Aug 19 at 15:34
9
$\begingroup$

None of your suggestions are good. You should use all the data.

The concern with collecting too much data is nothing to do with the generalizability of the study, but the impact to participants. In a randomized controlled trial, you might be doing a surgery or administering a drug which may be harmful, so you owe it to prospective study participants to analyze the results at the soonest available point. Even then, there can be logistical issues that cause a few additional patients to be treated here and there. In all scenarios, it is best to be honest about your design and what happened.

In your design, you had a survey that was opened for a period and you were flooded with results. This is a kind of low latency convenience sample. In my experience, these designs are received fairly negatively. This design (regardless of the $n$) has a lot of issues. It will prompt a lot of questions from reviewers. If they are good reviewers, they will be equally as dubious of your design whether you got 500 or 5,000 responses. Knowing that you beat initial sample size projections is almost a moot point.

Again, irrespective of the $n$, to correctly analyze the data, I would devote a lot of language to the magnitude of effect knowing that your study and results are highly calibrated to detect results which are (potentially) clinically not significant or relevant.

Improve this answer
$\endgroup$
2
  • 3
    $\begingroup$ +1 for "equally as dubious of your design whether you got 500 or 5,000 responses." For an intuition of how fatal a non-representative sample can be to stat testing, imagine trying to predict the coming US election by surveying the attendees at the ongoing Democratic National Convention. You could get 50000 responses there and P values barely distinguishable from zero, but you still won't have a better sense of the national mood than if you hadn't bothered! $\endgroup$
    – Josiah
    Commented Aug 20 at 7:24
  • 1
    $\begingroup$ @AdamO Could you please provide more information on "low latency convenience sample", please? I have never heard of it (convenience - yes, low latency con.sam. - no). I typed it into Google and there's nothing interesting. Do you have a paper on it for example? Fortunately, we became suspicious of how fast the sample size grew, so we did another research. This time around we asked a nationwide research panel for a sample. $\endgroup$
    – Lil'Lobster
    Commented Aug 21 at 10:17
2
$\begingroup$

Some other reasons to reject "excess" data I can imagine would be prohibitively high cost to store and process them. I guess this is not the case.

I don't know how your system determines the minimum sample size, but it should be emphasized it's minimum. Higher sample size reduces your statistical uncertainties and lets draw conclusions with higher confidence. It needs to be taken into account that there are likely effects contributing to the uncertainties, which don't decrease with growing sample size, though.

But most importantly, higher sample size let's you perform a more differential analysis. E.g. compare answers of people of different age. In particular I would like to comment on the following point from the highest-voted answer:

concerns about manipulation (in the sense that someone might think you peeked at the data after 500 respondents, didn't like the answer, let it keep going and then stopped when by chance you finally got something you liked - I'll assume this doesn't apply).

Higher sample size let's you compare various subsamples and actually demonstrate that your dataset is free of some potential biases. E.g. you can show that the answers 1–500 are consistent with 501–1000, 1001–1500 etc. Or you may discover that they are not, e.g. because the population of the people submitting the answers varies with the day of the week, and cutting the experiment after a few days was too early.

Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ A larger sample size does not allow to say that the sampling is free of bias. To take an extreme example, if your survey is only online, and supposed to be a random sample from the world population, it will be biased no matter the sample size, no matter for how long it is conducted, and not matter if all Internet users are miraculously equally likely to participate: see how many people in the world do not have an internet access, or how many people do not even know how to read or write (let alone language barriers, handicap, people in zones of war or natural disaster, dictatorships, etc.). $\endgroup$
    – J-J-J
    Commented Aug 21 at 8:35
  • $\begingroup$ @J-J-J I think we can agree that every sample is biased in one way or another ;-) I would side with user1079505 There is an interview on YT with Andrew Gelman about research mistakes who shows that sometimes conclusions like "people give more money at night" are wrong because of the randomization problem. Maybe my point here is not strictly about sampling free of bias, but the idea in the answer about checking the batches of participants is interesting. The first respondents may have been exceptional, very different from the population as a whole. $\endgroup$
    – Lil'Lobster
    Commented Aug 21 at 10:42
  • 1
    $\begingroup$ @J-J-J right, I stated this too strongly. What I wanted to write is that you can test for some potential biases, like the one mentioned in the quoted text. $\endgroup$
    – user1079505
    Commented Aug 21 at 21:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.