# Resolving Power in an AB test

I have been doing AB tests for quite a while. I use either a Fisher Exact Test or a Welch t-test depending on if my data is expected to be binomial or Gaussian, respectively. These produce p-values which I am happy with.

I also include the difference in the statistical means which is essentially the expected size of effect.

What I would like to do better is to compliment this with something like resolving power. This would be the size of effect I would be able to see a difference in means for based on my samples. This would be related to my False Negative rate I think.

My first thought is to calculate confidence intervals for each mean and state the gap between them. ie "at 95% CL there is more than a x% difference between A and B". However, I think this is just a restatement of my p-values and my False Positive rate. I want to be able to say "An effect of size X or larger would have been seen with 95% confidence"

Any pointers helpful. The more math the better.

You may be looking for a form of power calculation (perhaps based on raw effects rather than standardized ones).

[This shows a typical situation in calculating power. The top plot is the circumstance under the null, from which you have a rejection rule; the other plot is the situation under the alternative, using the same rejection rule -- you calculate the probability of rejection for a specific situation under the alternative.]

Indeed it sounds like you're describing a post-hoc power analysis, but as you say, this will be a function of the p-value. In addition some aspects of post hoc power analysis can be counterintuitive. They're somewhat controversial and should be used with caution.

However, an a priori calculation of power can be highly informative at any point -- not just in choosing sample size (which is what it's often used for), but in understanding, for example, how to interpret a rejection (if you have very low power but reject, you might wonder whether there's really anything going on there at all)

If you can specify the size of the difference you're looking to pick up you can compute the probability of rejecting the null at a given sample size.

Indeed you can calculate power curves which show the probability of rejection at every size of difference from the null (for a given sample size), or the probability of rejection at each sample size for a given effect.

(This power curve is generated by calling a function in R that gives power for the ordinary two-sample t-test]

Many questions on site deal with calculation of power.

For a variety of more complicated tests explicit expressions for small sample power may not be easy. In large samples you can often get asymptotic expressions that will work well. In the Welch t-test if you specify the ratio of variances, the two sample sizes and the how many of one of the standard deviations the two means are apart, you can calculate an asymptotic expression for the power.

In cases where algebraic calculation is difficult, simulation can give rejection rates at any specific alternative of interest, or across a sequence of alternatives. There's an example in R here (under Power of the t-test).

Here's an example power curve from a bit of research I was working on recently:

The points are simulated power values. This is for a widely used test.The situation being considered would be fairly representative in the particular application under consideration. The test has a problem in this particular circumstance, because even though it's at a big sample size, the test has low power -- particularly on the left side. (It also doesn't get very close to the required significance level and is biased -- the rejection rate is lower than $\alpha$ for a range of values where the null is false. This sort of thing would not be a substantial issue for your circumstances.)

[In this case, the fact that the power isn't quite high when the parameter (delta) is greater that 0.1 or less than -0.1 would be alarming; it will likely be a shock to the people who use the test in practice, because a difference of that size would be financially very important. The fact that they have little hope of detecting that large a difference with a widely-used test matters.]

• If I understand your plot correctly it is probability of rejecting the NULL as a function of the difference of means. This is exactly what I would like to calculate after data taking. I would be happy with the 95% confidence point but curves are always better. I suspect that you are in a special case since it is not minimized at 0. I would really prefer to avoid simulations. In the binomial or Gaussian case is there an analytic form? Commented Jun 7, 2017 at 4:30
• The null is certainly the only thing you should speak of rejecting. One can do exact calculations in the Gaussian case with a two sample t-test, but in the Welch case there's no simple analytic expression. It might be possible to use integration (perhaps numerically) to find the probability, but simulation is fast, easy and can be done to any desired level of accuracy. (NB... I just made some additions relating to the distinction between a prior and post hoc power calculations that may be relevant to you.) Commented Jun 7, 2017 at 4:51
• In the two sample binomial I presume you're referring to using the Fisher exact test, in which case you're conditioning on the margins and actually using hypergeometric probabilities; depending on what you're saying the situation is, it may be possible to do something... but again, I'd probably just be simulating. Commented Jun 7, 2017 at 4:51