We have analyzed several past online A/B tests and revealed that some of them have SRM at a statistically significant level (p<0.001). Example: we had 50%/50% traffic allocation between control and test variations. However, we ended up with the number of users below:

Control: 130631
Test: 133192

Some of the past tests had a discrepancy of 2% in the number of users between control and variation. Some had 5% or 10%.

Is there a percentage of discrepancy that can tolerate if a Sample Ratio Mismatch is detected?

Like less than 2% is tolerable, and we can still trust the results of our A/B tests. If more than 2% we can't do that.


1 Answer 1


The percentage of discrepancy doesn't tell you much on its own. For example:

  • prop.test(c(25, 20), c(50, 50)) has a 10 percentage-point discrepancy (50% vs 40%) but the p-value is clearly non-significant ($p=0.42$)
  • prop.test(c(2500, 2400), c(5000, 5000)) has a 2 percentage-point discrepancy (50% vs 48%) but the p-value is significant at the 5% level ($p=0.048$).

Therefore, what matters is the combination of the discrepancy and the sample size.

If the p-value of a SRM test is very small (e.g. $p<0.001$), that tells you that it would be extremely unlikely to get a discrepancy like the one you got (or larger) if your experimentation system was working properly. I would certainly be worried with such a small p-value.

On another note, if you are running many SRM tests on past A/B tests, you probably want to adjust for multiple comparisons. Otherwise, even if all your tests were correctly set up, you would still expect to get a $p<0.05$ approximately 5% of the time (false positive).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.