A webinar the other day by an a/b testing company had their resident "Data Scientist" explain that you should validate your results by re-running the experiment. The premise was, if you select 95% confidence, there is 5% (1/20) chance of a false positive. If you re-run your experiment with the same constraints, now there is a 1/400 (I'm assuming they determined this as 0.05^2 = 1/400)

Is this a valid statement? (ie, "run twice, two statistical significance wins = 1/400 probability of false positive")? Would it have been a better approach to increase your significance level?

From a business standpoint, the concern I have is by re-running the experiment, you are exposing more users to an inferior page (treatment), and thus losing out on potential sales.

  • 2
    $\begingroup$ Hi John, welcome to Stats.SE! If you're satisfied with either of the answers, you should accept one of them, or provide more clarifying questions about what you're looking for. $\endgroup$ Mar 22, 2014 at 23:50
  • $\begingroup$ John, I suspect the real issue concerns the context. It is rare that people will dedicate resources to learning only one thing at a time: they want to make the most of their data, for good reason. That means that each dataset will be used for multiple tests. Moreover, sometimes the tests are post hoc: they were inspired by patterns seen in the data. In such cases the tests do not actually have the desired 95% (or whatever) confidence and replication is essential. So: what precisely do you mean by "experiment"? The answer hinges on that little detail! $\endgroup$
    – whuber
    Mar 26, 2014 at 21:53
  • $\begingroup$ About experiment repetitions and significance values, check this XKCD comic: xkcd.com/882 After reading that, check whuber comment above. $\endgroup$ Mar 27, 2014 at 13:23
  • $\begingroup$ whuber: sorry for lack of detail, I'm referencing website optimization, so an example experiment would be trialing two versions of my homepage, with a 50/50 split of users to each. $\endgroup$
    – John
    Mar 27, 2014 at 13:40

2 Answers 2


Ignoring the probabilities of a false positive for the moment, I would look at it like this:

  1. If you run the experiment twice an get the same result, you have no idea whether there was two true positive results or two false positive results in a row.
  2. If you run the experiment twice and get two different results, then you do not know which is the true positive and which was the false positive result.

In either case you should then run a third experiment, just to be certain. This maybe fine for experiments that are relatively inexpensive, but where the cost is potentially high (like losing customers) you really need to consider the benefit.

Looking at the probabilities, the first time you run the experiment, there is a 1/20 chance of a false positive. The second time you run the experiment there is still a 1/20 chance of a false positive (think of it as rolling a die where each roll has a 1/6 chance of obtaining a certain number). There is only a 1/400 chance of having two false positives in a row.

The real issue is to have a well defined hypothesis with stringent procedures, and to have a sample size, level of error, and confidence interval you can live with or afford. Repetition of the experiment should be left to exploring

  1. customers over time
  2. changes made by the organisation
  3. changes made by the competition

rather than second guessing results. Although explaining this to managers is easier said than done.

  • $\begingroup$ mjc, thanks so much for the comment - this is exactly what I was looking for. $\endgroup$
    – John
    Mar 27, 2014 at 13:41

Yeah that statement is correct, assuming your experiment is ideal. But getting an ideal experiment is way way harder than this sentiment gives credence. "Real world" data is messy, complicated, and hard to interpret in the first place. There's tremendous room for flawed analysis, hidden variables (there's very rarely "the same constraints"), or miscommunications between a data scientist doing their job and a marking exec doing theirs.

From a business standpoint ensure good methodology and not being overconfident in results; a trickier challenge than you might think. Once you get those down, then work on that 5%.

  • $\begingroup$ Thanks, that answers the first question. What about the second question: "Would it have been a better approach to increase your significance level?" Just doing a quick simulation in R (keeping same effect size and power, only changing significance value) I could collect ~4.8% less data by simply choosing 97.5% significance, rather than running 2X experiments at 95% significance. I should clarify - when I ask "Would it have been better.." I mean, could I achieve the same end-result by collecting less data. $\endgroup$
    – John
    Mar 24, 2014 at 18:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.