5
$\begingroup$

I'm looking at data from experiments that have been run to look for a difference between proportions, 1-tailed.

In many of these it has been found that there is a significant difference between the two proportions (e.g., effect size of 3%, p<0.05)

However, I have run a post-hoc power analysis, inputting the effect size (as it was observed), desired power (0.8), significance (0.05), and it suggests that a sample size notably larger (for example 3x larger) than that which was used is in fact needed to run the test at this power / significance level.

The problem I have is interpreting this. If the power analysis had been done before and we had just happened to input the actual resulting effect size - the required sample size would have shown as this larger N, so by finding significance at a lower N, how should I interpret the result? How much confidence (if any) is lost?

As an aside: I know it is advised against to do post-hoc power analysis with the observed effect size, but I'm just trying to validate the quality of the test and how confident I can be in the significance that was found... as described above, hypothetically the exact same power analysis could've been done identically beforehand and would have suggested this higher N that was ultimately not reached.

I just don't know what implications this has for the effect size and p<0.05 that's then found in the data when the sample size < required N.

$\endgroup$
  • $\begingroup$ What variance estimates did you supply to the power calculations? $\endgroup$ – whuber Jul 16 '14 at 17:26
  • $\begingroup$ To be honest there was no variance input to the power calculation - this is a custom online tool provided for us to run these experiments. The inputs were the expected control proportion rate (predicted from past experiments), the minimum effect size, the power & significance levels, and choice of 1- or 2-tailed. From this a sample size was proposed... $\endgroup$ – Antony C Jul 17 '14 at 8:19
  • 1
    $\begingroup$ The issue I would have raised earlier, if I hadn't been focused on other issues, is whether a 1.83% difference is even meaningful in terms of practice. After all, the goal of statistics is to find the truth, within a reasonable range of uncertainty -- not to find asterisks. Maybe this study is bigger than it needs to be! Anyway, Antony, you may be getting tired of all this, so maybe we should move this elsewhere if we want to keep discussing. $\endgroup$ – rvl Jul 29 '14 at 18:05
  • 1
    $\begingroup$ @whuber Just to clarify, this was not a test stopped as soon as significance was found. The power study was done afterwards though to get an understanding of how close the sample size that HAD been gathered was to the recommended sample size, given the effect size noted, and the implications (if any) on validity of the result and how strongly the test significance level holds. Apologies to you both (the comments are interesting and appreciated) but I'm still not sure if I'm totally clear on this. $\endgroup$ – Antony C Aug 7 '14 at 12:17
  • 1
    $\begingroup$ Obviously I cannot speak for @Russ, but I believe we agree on all major points. The principal one is that a post hoc power study has no bearing on your current results. Another is that the information from these results would be useful for conducting a power study if you (or other researchers) wished to conduct another study (for instance, an independent replication of this one). A third is the useful and valid admonition to pay attention to the effect size and not get lost in concerns about achieving statistical significance. $\endgroup$ – whuber Aug 7 '14 at 13:57
5
$\begingroup$

I am among the voices that advises against post hoc power, as often as I get the chance. It's a silly thing to do, and doesn't add information. Power is the probability of rejecting the null hypothesis. So an interpretation of post hoc power is to answer the question "what is the probability that I rejected the null hypothesis?" Well, duh, you did reject, so it's equal to 1.

That said, the most common practice of computing post hoc power is to ignore the outcome of the test and do a calculation based on the observed effect size. In most cases, if you have borderline significance (commonly, $p$ just less than .05), then the post hoc power computed in this way will be about .50, though in really skewed situations it can be somewhat different. You'll get higher or lower post hoc powers than that depending on how much $p$ is less, or more, respectively, than the stated significance level.

Put another way, it stands to reason that if you were to repeat exactly the same study, there's about half a chance that your results will be less significant than they were this time. To be 80% confident that the next study will also show significance, you need to up the sample size. But none of this makes any sense unless you truly intend to repeat the study, because power analysis is inherently prospective.

Returning to the present, you already have data to back you up in confidently stating that the two proportions differ. That's enough - don't look a gift horse in the mouth.

$\endgroup$
  • $\begingroup$ +1 I would like to remark that the penultimate paragraph may mislead some readers. The "half a chance" is kind of a Bayesian conclusion that implicitly presupposes some prior distribution of the effect size. If that prior were focused near the null--as it should be--then we might actually conclude there is a very high chance that a replication of the study would have a smaller effect size. Another way to approach this question (without invoking any prior) would be to construct a prediction interval for the outcome of a future study. $\endgroup$ – whuber Jul 29 '14 at 12:09
  • $\begingroup$ Well, I can assure @whuber that I was not wearing my Bayesian hat when I made that statement (if I had, I might have been more careful!) I was just trying to say that the one study we have done is most likely in the middle of all potential studies of identical design, under identical conditions. So if I had to guess, I'd say half such studies would have $p$ less than what I observed, and the other half would have $p$ greater. $\endgroup$ – rvl Jul 29 '14 at 15:52
  • $\begingroup$ @Russ thanks for your feedback. Trying to piece it together in my head as a statistical novice :) Just on the last point, where you say the result showing a difference in proportions is enough. I'm still wrestling with the idea of test validity, the minimum sample size to detect effect sizes at the required significance level etc. As known, if a test was stopped as soon as significance was reached this would be ill-advised, increasing the chance of the result not reflecting the 'reality'. Doesn't failing to reach a recommended sample size for the detected effect size not have the same effect? $\endgroup$ – Antony C Aug 7 '14 at 12:29
  • 1
    $\begingroup$ Antony, if you had stopped the study when the desired significance was reached, then you'd be on thin ice like you say. But my understanding is that you collected the data and then analyzed it using standard, accepted techniques. Post hoc power is an attempt to re-analyze the same data and is completely unnecessary (plus it is not an accepted, standard technique in the eyes of most statisticians). Power is for prospective use only. $\endgroup$ – rvl Aug 7 '14 at 13:05
  • 1
    $\begingroup$ There is a huge difference, because ethics and integrity matter. When you use statistical methods, you always have a chance of being wrong. Your chance is controlled at .05 or whatever you specified, and the cheater's risk is much greater. If you're uncomfortable with being only 95% sure of yourself, by all means get more data. But otherwise, you were lucky enough to detect a difference that mattered, and you did everything right. So why not move forward in your research and not wring your hands about your past work? $\endgroup$ – rvl Aug 12 '14 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.