Should I bother that t-test power is relatively small when H0 is already rejected?

Question

I have two quite small samples ($n=7$ for each) and I get two-sample t-test power for them 0.49, which is relatively small (calculated by R power.t.test()). However, the Welch Two Sample t-test gives p-value of $0.032$ here, and so the null hypothesis can be rejected.

Now should I bother that the power is small?

My understanding is that power = $1 - \beta$ where $\beta$ is the probability of Type II error. Here it means that my test would fail to reject H0 in about 50 % of the cases when it should be rejected, so I should be worried especially when my particular samples couldn't reject H0. But in case of my particular samples it seems I'm lucky and my somewhat power-lacking t-test succeeded in rejecting, so I don't need to care about beta and I can enjoy being able to show a significant difference in the sample means.

Is my interpretation correct? Or do I miss anything important?

Answer 1

In a narrow sense, you are correct. Power is the chance to correctly reject a false null hypothesis, so you had a small chance but were able to do it anyway.

However, from a bayesian belief updating perspective, "reducing power implies reducing the belief shift that is warranted from observing a statistically significant result (McClelland et al. 2015)." Think of it this way: if I told you I surveyed 30,000 people from the general public and found that, contrary to sales figures, people tend to like Pepsi more than Coke, that would be very compelling. I found a result after studying 1% of a population (i.e. the US general public). It is likely to generalize to the larger population. If I surveyed 7 people and found the same thing, even if it was statistically significant, I wouldn't convince anyone. You can argue a lot of reasons for that (you can't get a representative sample, ANOVA/regression assumptions may not be met etc.), but what's important is that high power means highly persuasive (and you should be as critical or more of your results as those you are trying to convince). For the Bayesian mathematics as well as further explanation, you can check out any of the following.

Abelson, R. P. (2012). Statistics as principled argument. Psychology Press.

Brinberg, D., Lynch Jr, J. G., & Sawyer, A. G. (1992). Hypothesized and confounded explanations in theory tests: A Bayesian analysis. Journal of Consumer Research, 139-154.

McClelland, G., Lynch, J. G., Irwin, J. R., Spiller, S. A., & Fitzsimons, G. J. (2015). Median Splits, Type II Errors, and False Positive Consumer Psychology: Don't Fight the Power. Type II Errors, and False Positive Consumer Psychology: Don't Fight the Power (March 10, 2015).

Also, Ioannidis (2005) provides some compelling arguments to low power results reflecting type I errors even in the absence of p-hacking and other biases that often result from low power (and the paper is open access in case you don't work for a university or something similar!).

Answer 2

It depends on how the power analysis was conducted. Generally speaking, there is a relationship between four variables (alpha, power, the effect size, and $N$) such that if you stipulate any three you can solve for the fourth.

Typically, when people run a power analysis after analyzing their data, they do a post-hoc power analysis (hence the name), which means they plug in their value for alpha, their observed effect size, and their $N$, to solve for power. You certainly don't have to do it that way (you could do it beforehand with a theoretically suggested effect size and the $N$ you know will be available to you), but if you do, the results are largely foregone. Specifically, if your result was significant, the power will be $>50\%$, and if your result was not significant, the power will be $<50\%$.

That doesn't seem to be quite what you found. One possibility is that there is a subtle discrepancy between the methods used in your test and those in the power analysis. This is suggested by the fact that your observed $p$-value is close to $.05$ and your computed power is close to $50\%$, even though they are on different sides of the line. Another possibility is that you used a slightly different effect size from what you found.

So, "should [it] bother [you] that the power is small?" Yes and no. If you did a traditional (invalid) post-hoc power analysis, you were necessarily going to get values like that—the exercise was completely uninformative. On the other hand, if we take the power analysis seriously, a significant effect with a low powered setup basically means that your observed effect has to be biased to be larger than it really is, so you should be less trusting of the results.

Answer 3

Denote $p$ the $p$-value of your test (as a random variable) and fix some $\alpha$. Call a test result significant or positive when $p \leq \alpha$. We have $P(p \leq \alpha \,|\, H_0) \leq \alpha$. Moreover, let $\beta$ be such that $P(p > \alpha \,|\, H_1) \leq \beta$. Then $1-\beta$ is the power of the test.

Treating $H_0$ and $H_1$ as (complementary) events, Bayes' theorem gives: $$\frac{P(H_1 \, | \, p\leq\alpha)}{P(H_0 \,|\, p\leq\alpha)} = \frac{P(p\leq\alpha \,|\, H_1)}{P(p\leq\alpha \,|\, H_0)} \cdot \frac{P(H_1)}{P(H_0)} \geq \frac{1-\beta}{\alpha} \cdot \frac{P(H_1)}{P(H_0)}$$ This shows that the post odds for $H_1$ are a scaled version of the prior odds, with the strength of the scaling in favor for $H_1$ increasing with $1-\beta$. This means we learn more from a positive test when $1-\beta$ is large.

For further illustration, look at confidence intervals (CI). One may argue that larger sample size will make the CI more narrow and thus, if the test was significant for a smaller sample, it will also be significant for the larger sample. However, also the location of the CI can shift when we include more data in our sample, potentially making the result non-significant. It is also conceivable that the larger sample will have a much larger standard error and thus the CI will become wider in fact. One could say that a larger sample size gives the facts more opportunity to prove themselves.

There has been some interesting discussion lately about the interpretation of $p$-values, see, e.g.:

[1] Colquhoun, "An investigation of the false discovery rate and the misinterpretation of p-values", Royal Society Open Science, 2014

[2] Colquhoun, "The Reproducibility Of Research And The Misinterpretation Of P Values", 2017, http://www.biorxiv.org/content/early/2017/08/07/144337

[3] "What would Cohen say? A comment on $p < .005$", https://replicationindex.wordpress.com/2017/08/02/what-would-cohen-say-a-comment-on-p-005/

Concerning your particular result, I am not qualified to judge it. Using only your $p$-value and the classification from [2], it is between "weak evidence: worth another look" and "moderate evidence for a real effect".