# How can power and effect size help me interpret my A/B test results?

To give an idea of my use case, I'm performing an A/B test using a two-sample t-test to compare the mean between a control and a treatment sample.

I set $$\alpha=0.05$$. I set my effect size $$d=0.01$$ and my power=0.80. Then, using a power solver, I get $$n=156978$$. I understand that power, effect and alpha help me define the sample size.

Then, I collect the data and compute the p-value. Now I can make a decision: if p-value<$$\alpha$$, then I can reject $$H_{0}$$; if not, then I fail to reject $$H_{0}$$. Clearly, $$\alpha$$ has a purpose: decide the outcome of my test.

However, what is the purpose of the power and effect size?

Power and effect size don't impact the p-value: I could arbitrarily increase or decrease both power and effect size (and keep constant $$n$$); it would not change the p-value, therefore, no impact on the test outcome.

So maybe they can help interpret the test results (e.g. give me some probability that my test is correct or something?). The power=0.80 tells me there is an 80% probability of rejecting $$H_{0}$$, assuming that the effect size I chose is the true effect size. But in real life, we can never know if that is the case (if we knew we would not be doing a test in the first place), so how is it helpful in any way?

Question: I'm struggling to understand how to interpret my test results in light of the power and effect size I chose. Other than "I need them to plug into my power solver". Indeed, they can help me interpret my test results in one way or another.

• At the outset you write (correctly), "power, effect and alpha help me define the sample size." Why then do you need to ask "what is the purpose of the power and effect size" when you have already answered that question?
– whuber
Commented Jan 29 at 16:34
• The mistake is in thinking that the p-value tells you whether or not your theory is true. In fact the p-value tells you how much evidence you gathered to reject your null hypothesis. The power and effect size you chose tells you how hard you looked for that evidence. Commented Jan 29 at 16:34
• I think there's some confusion around the p-value, which depends only on the data itself and not the expected effect size or power, and the alpha level, which is the fixed p-value threshold at which you make a decision with particular Type I/II error rates. The data produces some p-value regardless of your expectations, the question is if that's sufficient evidence for your alpha and power levels. Commented Jan 29 at 16:48
• @NuclearHoagie I agree, p-value depends only on data. But I can see how it relates to alpha: you compare p-value<alpha to decide on test outcome. How does it relate to the power level? Commented Jan 29 at 17:03
• @KiwiKiwi It relates inasmuch as alpha and power relate. For fixed N and effect size, if you want to be more sure that you will not miss calling a real effect significant (lower Type II error/higher power), you must allow a higher chance of calling no effect significant (higher Type I error/alpha). The p-value is your level of evidence (a result of your N), how you use that evidence to make a decision depends on your prior beliefs (effect size), and the tradeoff of what types of errors you're willing to make (alpha and power). Commented Jan 29 at 17:22

Power and effect size together tell you the probability that you'll correctly find a significant result, if the true effect really is as large as the effect size.

You've correctly noted that the p-value you actually observe in any statistical test cannot depend on your desired power, or how big you think the effect is, it's only determined by the data itself. What you can control is the critical value of the p-value, that is, what p-value you need before you call something "significant". For some dataset of N subjects, you can compute a p-value - the question is, when do you reject H0? That decision directly depends on alpha, which is related to power.

How you actually use the p-value you find to make a decision of significant or not significant will depend on your prior beliefs about the effect size, and which types of errors (Type I or Type II) you are willing to make. For fixed N and effect size, if you want a high power to ensure you won't miss real effects, you'll naturally have a higher Type I error rate and high alpha, and will require very little evidence to claim a significant result. On the other hand, if you don't want to erroneously claim significance, you'll require lots of evidence to claim a significant result. The exact same p-value may or may not represent "significance" depending on your expectations and the implied relative impact of Type I and Type II errors.

Power, alpha, N, and effect size are all related, given any 3 you can figure out the last one. For a fixed N and alpha, power and effect size are correlated - if you believe the effect is large, you'll have high power to detect it if it exists.

• I don't see how "a priori believe there is a very large effect size" implies "you'll naturally have a high Type I error rate and high alpha." Why should your belief about the magnitude of the effect determine the acceptable risk of not rejecting the null (in case the null just happens to be true)?
– whuber
Commented Jan 29 at 19:30
• @whuber I think I conflated a couple of ideas and miscommunicated the result. The general idea was that if you want to find a small effect, you need lots of data to call significance at some alpha and power level. If you don't have that much data, you need to either look for larger effect, or at a less stringent significance level, or both. They can indeed be adjusted separately, but testing for only very large effects may imply that small-effect tests would have had too high an error rate. Commented Jan 29 at 20:57

You wrote

I could arbitrarily increase or decrease both power and effect size (and keep contant n)

No, this is incorrect, unless no one else will look at your analysis. It's true that if you set a very large effect size, then the required sample size will be small. But people reading your analysis will not believe you. If you develop a drug for some sort of cancer and say that it will reduce one year mortality rate by 99%, well, prepare for skepticism. You, on the other hand, have set a very low effect size (although it is context dependent). If you can gather data cheaply and easily, that's not so much of a problem. But how often is that the case?

As to your question, well, no, not really. Once you have output, the power analysis sort of fades into the background. Your output will include test statistics, standard errors, p values, and confidence intervals. That's what you interpret after doing the test.

However, if you find that you can't gather the required sample size, and that you get non-significant results, then you can say "the test may have been underpowered." But then someone may ask "you did the power analysis, it said you needed an unrealistic sample, so why did you continue?"

I'll take an unpopular but correct stance. If you want to 'interpret' or understand, or do anything else beyond an automatic decision that would be relevant to acceptance testing of widgets, you should ignore the pre-test $$\alpha$$ and entirely once you have the data. The accept/reject decision of the Neyman–Pearsonian hypothesis testing method deals with the long run error rate properties of the method, not the evidential meaning of the data.

Sure, you can use $$\alpha$$ and power in pre-experiment planning, but if you want to know what the data say then the p-value is where it's at (or likelihood function, or a Bayesian analysis), not the all or none larger or smaller than $$\alpha$$.

You can read much more about it (and other useful things like expected p-value instead of power for sample size estimation) here: https://link.springer.com/chapter/10.1007/164_2019_286

Surely, they can help me interpret the results of my test, one way or another?

Power is something computed conditional on some assumption about the true effect size. If the effect size was at least as large as what you assumed, then the probability you would reject the null correctly is the statistical power.

Once the study is run, power says nothing about what was actually observed.

Power does imply something about the effect you end up detecting. Assume the alternative you've specified is true but your power is smaller. Then if you end up detecting an effect, then that effect will be much larger than the truth. Reason being is you've designed your test in such a way that only those detects much larger than the truth end up being statistically significant.