1
$\begingroup$

This is actually a true story.

While planing a study, my work colleague did a power analysis for an intended effect size and a priori power of 80%. It turned out that the required sample size was one hundred observations per one group (n = 100), so he needed two hundred overall (N = 200). By now everybody figured out that was about Student's t-test (you're right). Then, the colleague contacted some data collection company and asked them for 200 observations in his study. It was paid in advance.

What the company did was to accidentally collect two hundred observations per one group (n=200). Twice as much as it was indicated in power analysis. Probably they misunderstood or probably he didn't explain clearly. Anyway, they kindly apologized for the inconvenience and didn't charge him for the additonal data. But he was left with those four hundred (N=400), which according to power analysis created an overpowered study. He scratched his head and said that he was gonna randomly select one hundred observations in each group to retain intended power of 80%.

I'm not sure if what he planned was a proper choice, so I'd like to ask you what do you think of it all? Should he stick to his a priori power-analysis results and randomly select participants from already collected data? Lower alpha level below usual 5%? Or there are other options?

Besides it all seems to me as if more data is something bad.

Improve this question
$\endgroup$
  • 2
    $\begingroup$ I wasn't aware that it was even possible for a study to be overpowered. $\endgroup$ – user234562 Aug 1 '20 at 12:06
  • 1
    $\begingroup$ If his objective is (cynically) to run a study in the hope that it will, by chance, not detect the effect of interest, then your colleague will be upset that the study is over-powered. Given the analysis is with a t-test, there's no added cost to analyzing the larger dataset (which, if there were, could explain an investigator's reluctance to use all the data). Assuming the data were properly collected, I cannot think of any other possible reason not to use all the data. $\endgroup$ – whuber♦ Aug 1 '20 at 12:49
  • 3
    $\begingroup$ Now, how to use all the data is another question. For instance, it is tempting to run the original study on the first half of the data and then check the results with the second half. (But there are better alternatives--this merely illustrates the possibilities that are opened up.) Suffice it to say that this is a windfall that can be exploited in many ways. $\endgroup$ – whuber♦ Aug 1 '20 at 12:52
  • 1
    $\begingroup$ @user332577 To put it amusingly... Recently a new form of phobia is observed among research workers. They fear that if a sample size is too high, everything will be statistically significant. $\endgroup$ – Lil'Lobster Aug 1 '20 at 14:03
  • $\begingroup$ @Lil'Lobster :) That phobia is misplaced since it borders on one of the limitations of tests of significance, the recognized fact that, given enough data, everything will be statistically significant. Hence, the recommended reversion to examining effect sizes as the better, more appropriate metric of whether or not something matters. $\endgroup$ – user234562 Aug 3 '20 at 14:06
1
$\begingroup$

The purpose of doing a power analysis is to get your minimum samples size. It generally costs time and money to get samples.

What is wrong with an "over powered" study? Nothing. There is something right with it, though. The results are more likely to be representative of the true population.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.