Validate web a/b tests by re-running an experiment - is this valid?

Question

A webinar the other day by an a/b testing company had their resident "Data Scientist" explain that you should validate your results by re-running the experiment. The premise was, if you select 95% confidence, there is 5% (1/20) chance of a false positive. If you re-run your experiment with the same constraints, now there is a 1/400 (I'm assuming they determined this as 0.05^2 = 1/400)

Is this a valid statement? (ie, "run twice, two statistical significance wins = 1/400 probability of false positive")? Would it have been a better approach to increase your significance level?

From a business standpoint, the concern I have is by re-running the experiment, you are exposing more users to an inferior page (treatment), and thus losing out on potential sales.

Answer 1

Ignoring the probabilities of a false positive for the moment, I would look at it like this:

1. If you run the experiment twice an get the same result, you have no idea whether there was two true positive results or two false positive results in a row.
2. If you run the experiment twice and get two different results, then you do not know which is the true positive and which was the false positive result.

In either case you should then run a third experiment, just to be certain. This maybe fine for experiments that are relatively inexpensive, but where the cost is potentially high (like losing customers) you really need to consider the benefit.

Looking at the probabilities, the first time you run the experiment, there is a 1/20 chance of a false positive. The second time you run the experiment there is still a 1/20 chance of a false positive (think of it as rolling a die where each roll has a 1/6 chance of obtaining a certain number). There is only a 1/400 chance of having two false positives in a row.

The real issue is to have a well defined hypothesis with stringent procedures, and to have a sample size, level of error, and confidence interval you can live with or afford. Repetition of the experiment should be left to exploring

1. customers over time
2. changes made by the organisation
3. changes made by the competition

rather than second guessing results. Although explaining this to managers is easier said than done.

Answer 2

Yeah that statement is correct, assuming your experiment is ideal. But getting an ideal experiment is way way harder than this sentiment gives credence. "Real world" data is messy, complicated, and hard to interpret in the first place. There's tremendous room for flawed analysis, hidden variables (there's very rarely "the same constraints"), or miscommunications between a data scientist doing their job and a marking exec doing theirs.

From a business standpoint ensure good methodology and not being overconfident in results; a trickier challenge than you might think. Once you get those down, then work on that 5%.