2
$\begingroup$

I ran an A/B test on a website. I get a p-value of 0.45: not statistically significant. There is a 4.5% difference in variation A vs. the control (B). Assume the confidence interval is -8%

So, here is the conclusion that I'm making, though I know it's flawed: "Well, the result was not statistically significant. BUT, if I'm tasked with making a decision between variation A and control, why wouldn't I go with the control at 55% significance and 4.5% lift?"

What is inherently flawed with this interpretation? What would be the proper way to assess this kind of result?

Improve this question
$\endgroup$
11
  • $\begingroup$ Sorry, let me clarify: let's say the P value was .45, and the outcome is conversion rate for the website (purchased a product). Does that help? $\endgroup$ – optimizer Apr 10 '18 at 14:22
  • 1
    $\begingroup$ The experiment lacks the statistical power to clearly ascertain whether A or B is better. What to do? Following Bayesian decision theory, your choice of A or B would also incorporate (i) prior beliefs about A or B and (ii) your preferences over risk. $\endgroup$ – Matthew Gunn Apr 10 '18 at 14:34
  • 1
    $\begingroup$ Thanks Matthew. So I think that makes sense. So, basically, here is what we can conclude: There is little evidence that the perceived results are due to the experiment. Instead, the perceived difference is due to chance. So, if we see a 4.5% lift, we're running the risk (albeit, seemingly, a lower risk because we see a p value of .55, compared to a p value of say, .9), but a risk that the page we roll out is in fact NOT a winner, and is in fact a loser as seen in the confidence intervals. Am i right in that interpretation? $\endgroup$ – optimizer Apr 10 '18 at 14:38
  • 4
    $\begingroup$ The classic, frequentist interpretation of your results is that if there were no difference between A and B, you'd be highly likely to see a result like this. To talk about the probability of effect A being higher than effect B, you need to adopt Bayesian viewpoint and add prior beliefs over effect A and effect B. $\endgroup$ – Matthew Gunn Apr 10 '18 at 14:43
  • 1
    $\begingroup$ Sal, thanks for the response. Can I answer your rhetorical questions? There are no practical costs to switching between A/B. Theoretically, it would be bad practice to switch, so we'd only keep one of the variations. 4.5% lift is material if significant, but it didn't go significant, so the lift isn't accurate. An extended test CAN be run - perhaps that's something else I can use to staple the point. $\endgroup$ – optimizer Apr 10 '18 at 15:02
2
$\begingroup$

Imagine B is the current system with a clickthrough rate of 3%. You test A on 1 person and observe 100% clickthrough. Should you change the whole website to A?

Why might we choose B even if A performs better in the data:

  1. We have prior beliefs that B is better (and our experimental results aren't strong enough to overturn those prior beliefs).
    • Example: Let A be yellow, blinking text. I have strong priors that blinking text is worse in 999/1000 cases; it will take powerful evidence to convince me blinking text is better (rather than a spurious result). Similarly, if a disease is more rare than a test is accurate, most positive test results will be false positives.
  2. We are risk averse and measure B more precisely than A.
    • Example: B is the extensively tested drug ibuprofen and A is a new, seemingly effective but potentially deadly painkiller. Even if we estimate A as better, we wouldn't deploy it widely until we're confident A is consistently safe.

Applying Bayesian decision theory, point (1) treats the parameter as a random variable and brings prior beliefs into the analysis. Point (2) is that we may be risk averse (which can be formalized using expected utility).

Additional complications:

  • Adjustment costs (eg. implementation costs or user learning costs) would raise the bar for making a change. For example if the change is irreversible, the hurdle to change may be especially high.
  • In a multi-period setting, making the the change has an additional payoff in that we get to learn more about the change (which may have value).
Improve this answer
$\endgroup$
5
  • $\begingroup$ This is solid reasoning. I think you're ultimately saying that there IS no inherit problem with making that decision if you're aware of the risks and adjust for those using bayesian interference. $\endgroup$ – optimizer Apr 10 '18 at 16:16
  • $\begingroup$ @optimizer I think that's right. The big concern is that people people misinterpret a 51% probability that A is better as a 100% probability that A is better. (eg. when candidate B narrowly wins even though candidate A had a statistically tiny lead in the polls, people often say, "The polls were wrong!" even though the polls were actually right in that the election was close.) $\endgroup$ – Matthew Gunn Apr 10 '18 at 16:40
  • $\begingroup$ @optimizer Basically, you learned from your experiment that effect sizes are probably small relative to your measurement error: your data doesn't tell whether A or B is better. Thus the decision of A or B should be mostly based on factors other than the data. $\endgroup$ – Matthew Gunn Apr 10 '18 at 16:43
  • 1
    $\begingroup$ "Thus the decision of A or B should be mostly based on factors other than the data" - I think this statement in your comment is of incredible importance as sort of a lay explanation of what it means to make Bayesian decisions in the face of indecisive data - might be worth incorporating explicitly into your answer. $\endgroup$ – Bryan Krause Apr 10 '18 at 22:05
  • $\begingroup$ This is exactly what I thought. So essentially, the data is saying there isn't enough evidence one way or the other to prove one is better than the other. There is inherently a risk in "picking" one versus the other. But, the data is simply saying "inconclusive" - you should pick a variation based on OTHER factors $\endgroup$ – optimizer Apr 11 '18 at 15:47
0
$\begingroup$

You are misinterpreting the meaning of a p-value. While I preferentially would advocate for a Bayesian method if money is involved, if a behavior is involved then I may take the Frequentist approach. If your p-value is $p<.45$ where $\mu_A=\mu_B$, this tells you that if the null is true, then there is less than a 45% chance of seeing a result as extreme or more extreme than what you saw. That means you have roughly a coin toss chance of seeing this data.

Under the three main interpretations of probability your inference would be as follows.

  1. Fisherian likelihoodist-- nothing was learned, it is time to go onto finding a different choice because there is no real evidence against the null
  2. Frequentist inference--the null is in the acceptance region so it makes no sense to treat option A or option B as different
  3. Bayesian inference--without more specific information no calculations can be performed, but knowing that the p-value is less than 45% tells me that there is a slight amount of evidence of one hypothesis over the other.

Your decisions though may be different.

  1. Frequentist decision theory--You accept the null and behave as if the null is true. This would imply, if you have a default such as B, then you should do B. However, in the absence of additional costs, you shouldn't care if you did A or B.
  2. Bayesian decision theory--you should base your acceptance or rejection criterion and the combination of the probabilities and the costs or utility of the choices. Since one option is slightly more probable than the other, if you control for costs, you should choose the one with greater probability of being true. If that choice is costly, you should factor in the cost of a false positive and a false negative. That relative cost should constitute your odds criterion for acceptance. If that criterion is passed, then you do the more costly choice.
Improve this answer
$\endgroup$
0
$\begingroup$

Without going Bayesian and bringing priors into the equation, a null result suggests that population parameters you infer about may be equal. Depending on the distribution, they may even be directionally reversed (ie, even if your sample statistic A > B, in the population B may be > A). If we were allowed to make decisions based only on the raw directionality of the sample statistics, there’d be no reason for a statistical test.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.