3
$\begingroup$

Im trying to understand how to accurately report results on AB tests

If my type 1 error is 5% and type 2 is 20% , can we have instances where the p-value is < 0.05 and power is < 0.8?

If so, how ? What does it mean ?

Is something statistical significant when p-value is under 0.05 or power over 80% (assuming alpha and beta are defined to be 5% and 20%)?

EDIT: Im performing my power analysis during an experiment. Im trying to give updates on how a change in a button colour is performing. And in that update I want to specify the power it has.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Are you at the stage where you are calculating the a priori power while designing your experiment (e.g. determining the sample size), or that where you have the responses back and are calculating the post-hoc power to see how likely you are to see an significant effect if what you observed is the true effect size? $\endgroup$
    – B.Liu
    May 20 at 15:36
  • $\begingroup$ @B.Liu the experiment is still running . $\endgroup$
    – Roger Steinberg
    May 21 at 0:58

3 Answers 3

Reset to default
6
$\begingroup$

Power depends on the $\alpha$-level, which you set, not on the p-value (which you calculate).

If you do a power calculation and determine that you have $80\%$ power to detect the difference of interest at the $\alpha$-level you've selected (often, but not necessarily, $\alpha = 0.05$), then that's the power. You don't even need to calculate a p-value for that to be the power. In fact, it might be common in certain fields not to have any data when the power calculation is performed, but to perform it in order to get funding for an experiment that will collect the data and result in subsequent calculations (such as p-values).

Improve this answer
$\endgroup$
3
  • $\begingroup$ i guess i wasnt too clear in my question sorry: 1. when people say it is statistically significant , does it mean that the p-value is smaller than their predefined alpha (e.g 5%) or above power threshold (e.g. 80%). 2. If you define alpha to be 5% and power to be 80%, is it possible that in the middle of the experiment you see p-value 0.002 and power 60%??/ $\endgroup$
    – Roger Steinberg
    May 21 at 1:02
  • $\begingroup$ probably also forgot to mention that this is an analysis im running WHILE the experiment is running and im providing updates along the way $\endgroup$
    – Roger Steinberg
    May 21 at 1:04
  • $\begingroup$ Yes, this is possible. A post-hoc power analysis can assume a different alternative parameter value than is observed in the data. For example, let's say you're doing a power analysis for a one-sample T test using the observed sample standard deviation. If H0 is that mu=0, your observed data may have mu=5, but your power analysis is calculated based on a population where mu=.1. $\endgroup$
    – clementzach
    May 21 at 1:26
1
$\begingroup$

Possible situations abound. Here a few: Suppose you are testing $H_0: \mu = 50,$ against $H_a: \mu \ne 50$ at significance level $\alpha = 0.01 = 1\%,$ using $n = 10$ observations from a normal population with $\sigma \approx 4.$

Then a power and sample size procedure (here from a recent release of Minitab) gives power values about 70%, 75%, 80% for alternative values $\mu_a = 54.82, 55.06,$ and $55.33,$ respectively.

Power and Sample Size 

1-Sample t Test

Testing mean = null (versus ≠ null)
Calculating power for mean = null + difference
α = 0.01  Assumed standard deviation = 4

Sample
  Size  Power  Difference
    10   0.70     4.81897
    10   0.75     5.05952
    10   0.80     5.32813

Corresponding points on the relevant power curve are shown below:

enter image description here

Note: A trivial example is to set the Rejection region to be so far out into the tails of the null distribution (say, $|T|\ge 30)$ that $\alpha \approx 0.$ Then the power (probability of rejection for the specified alternative) could also be very small--in particular less than 80%.

Improve this answer
$\endgroup$
2
  • $\begingroup$ So then why are we saying statistically significant when p-value is less than our defined alpha threshold if power is say 40%. If my false negative rate probability is 60% then we shouldnt state that it is statistically significant? Or does power only help determine the meaningfulness of what we've detected. I.e it is statistically significant at p <0.05 but not meaningful since we have a 40% chance of correctly rejecting the nullhypothesis. Do i even make any sense? lol $\endgroup$
    – Roger Steinberg
    May 21 at 17:23
  • 1
    $\begingroup$ @RogerSteinberg Think of it like any other situation where you get an unusual result. For instance, the low probability of winning the lottery is not important if I have the winning ticket. Yes, the probability is low, but I was lucky enough to be that one in a million. (There are issues regarding estimates of the effect size when studies have low power, but that is a somewhat different issue from rejecting or not rejecting a null hypothesis.) $\endgroup$
    – Dave
    May 21 at 17:31
1
$\begingroup$

If you observe a p-value equal to the alpha level, then you have roughly $50\%$ power for a true effect that equals the observed effect. Observations that are closer to the null hypothesis will not be significant and observations further from the null hypothesis will be significant. You have roughly 50:50 probability that these will happen (often the distribution is more or less symmetrical).

Here is an example for a normal distribution. From this question Report power if result is statistically significant

The alpha level is here $5\%$. We observe a p-value below $0.05$ and the power is below $0.8$.

example

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.