I ran an A/B test on a website. I get a p-value of 0.45: not statistically significant. There is a 4.5% difference in variation A vs. the control (B). Assume the confidence interval is -8%
So, here is the conclusion that I'm making, though I know it's flawed: "Well, the result was not statistically significant. BUT, if I'm tasked with making a decision between variation A and control, why wouldn't I go with the control at 55% significance and 4.5% lift?"
What is inherently flawed with this interpretation? What would be the proper way to assess this kind of result?