Interpretation of non-statistically significant Tests I ran an A/B test on a website. I get a p-value of 0.45: not statistically significant. There is a 4.5% difference in variation A vs. the control (B). Assume the confidence interval is -8%

So, here is the conclusion that I'm making, though I know it's flawed: "Well, the result was not statistically significant. BUT, if I'm tasked with making a decision between variation A and control, why wouldn't I go with the control at 55% significance and 4.5% lift?"
What is inherently flawed with this interpretation? What would be the proper way to assess this kind of result? 
 A: Imagine B is the current system with a clickthrough rate of 3%. You test A on 1 person and observe 100% clickthrough. Should you change the whole website to A?
Why might we choose B even if A performs better in the data:

*

*We have prior beliefs that B is better (and our experimental results aren't strong enough to overturn those prior beliefs).

*

*Example: Let A be yellow, blinking text. I have strong priors that blinking text is worse in 999/1000 cases; it will take powerful evidence to convince me blinking text is better (rather than a spurious result). Similarly, if a disease is more rare than a test is accurate, most positive test results will be false positives.



*We are risk averse and measure B more precisely than A.

*

*Example: B is the extensively tested drug ibuprofen and A is a new, seemingly effective but potentially deadly painkiller. Even if we estimate A as better, we wouldn't deploy it widely until we're confident A is consistently safe.



Applying Bayesian decision theory, point (1) treats the parameter as a random variable and brings prior beliefs into the analysis. Point (2) is that we may be risk averse (which can be formalized using expected utility).
Additional complications:

*

*Adjustment costs (eg. implementation costs or user learning costs) would raise the bar for making a change. For example if the change is irreversible, the hurdle to change may be especially high.

*In a multi-period setting, making the the change has an additional payoff in that we get to learn more about the change (which may have value).

A: You are misinterpreting the meaning of a p-value.  While I preferentially would advocate for a Bayesian method if money is involved, if a behavior is involved then I may take the Frequentist approach.  If your p-value is $p<.45$ where $\mu_A=\mu_B$, this tells you that if the null is true, then there is less than a 45% chance of seeing a result as extreme or more extreme than what you saw.  That means you have roughly a coin toss chance of seeing this data.
Under the three main interpretations of probability your inference would be as follows.


*

*Fisherian likelihoodist-- nothing was learned, it is time to go onto finding a different choice because there is no real evidence against the null

*Frequentist inference--the null is in the acceptance region so it makes no sense to treat option A or option B as different

*Bayesian inference--without more specific information no calculations can be performed, but knowing that the p-value is less than 45% tells me that there is a slight amount of evidence of one hypothesis over the other.


Your decisions though may be different.


*

*Frequentist decision theory--You accept the null and behave as if the null is true.  This would imply, if you have a default such as B, then you should do B.  However, in the absence of additional costs, you shouldn't care if you did A or B.

*Bayesian decision theory--you should base your acceptance or rejection criterion and the combination of the probabilities and the costs or utility of the choices.  Since one option is slightly more probable than the other, if you control for costs, you should choose the one with greater probability of being true.  If that choice is costly, you should factor in the cost of a false positive and a false negative.  That relative cost should constitute your odds criterion for acceptance.  If that criterion is passed, then you do the more costly choice.

A: Without going Bayesian and bringing priors into the equation, a null result suggests that population parameters you infer about may be equal. Depending on the distribution, they may even be directionally reversed (ie, even if your sample statistic A > B, in the population B may be > A). If we were allowed to make decisions based only on the raw directionality of the sample statistics, there’d be no reason for a statistical test. 
