0
$\begingroup$

Suppose we have an AB test running for enough time to observe an effect size of at least 3% on certain variable X. When checking the results, we obtain that the p-value is below the significance level (< 0.05) but the effect observed during the experiment is just 2%.

To me, it looks like we don't have enough statistical power to judge the results as significant even when we have a significant p-value. How do you judge these results in this situation? Can we say that the results are significant having that the observed effect size is smaller than the one the test was able to detect at least 80% of the time?

Is there any post-hoc analysis that could be ran on this situation?

I've been looking at some papers but some of them contradict each other when approaching to post-hoc analysis.

Thank you in advance for your help!

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Everything hinges on what "when checking the results" might mean. If, for instance, you mean performing a preliminary evaluation before the test has run for its planned time, this approach is invalid. If it has run for its planned time and the p-value indicates significance, then further power calculations are irrelevant. Which situation are you in? $\endgroup$
    – whuber ♦
    Aug 4, 2022 at 18:51

1 Answer 1

Reset to default
0
$\begingroup$

Maybe I misunderstand your post, but it sounds like:

  • Your results found a p-value below the significance level, leading you to reject the null hypothesis
  • You had conducted a power analysis, and determined your sample size expecting an effect size of 3%
  • Your effect size in the experiment was 2%
  • You are unsure what this means in terms of power

As a reminder, statistical power is 1 - $\beta$ where $\beta$ is the chance of a Type II Error: failing to reject the null hypothesis when the null hypothesis is false. Also known as a False Negative. Image Source: https://www.statisticshowto.com/probability-and-statistics/statistics-definitions/statistical-power/

Your significance test indicates you should reject the Null Hypothesis. You either have a Type I error (False Positive) or the correct result: finding an effect that actually exists. That is, you have a positive result, so having a Type II error (False Negative) is not a concern.

Are you sure you did the power analysis correctly?

Improve this answer
$\endgroup$
3
  • $\begingroup$ Check out this post for a longer discussion: stats.stackexchange.com/questions/377307/… $\endgroup$
    – stat_is_quo
    Aug 4, 2022 at 20:28
  • $\begingroup$ Thanks for your input! The power analysis was done before starting with the test, actually, to determine if after certain days of exposure we would see an effect size of 3%. The power analysis was done using a tool (unfortunately I don't have the details about it so it's kinda a black box for me) in which the input is the number of days that we could leave the test live, the effect size that we expect to see, the number of groups that we will have for the experiment and the proportion of individuals that will be in the control group. $\endgroup$
    – Brenda
    Aug 5, 2022 at 10:48
  • $\begingroup$ The output of the tool is a table and a chart, displaying for each day of exposure (in the x-axis), the statistical power that we are going to have for the test (in the y-axis). The tool uses some historical data available to determine the statistical power in the described way. In this particular case, before starting with the test, we check whether after 30 days of exposure, having 2 evenly split groups, we could detect an impact of at least 3% in certain metric IF there is one. After 30 days of exposure, we checked the results and obtain p value <0.05 but the effect size was just 2%. $\endgroup$
    – Brenda
    Aug 5, 2022 at 10:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.