How much of a discrepency is tolerable if Sample Ratio Mismatch is detected?

Question

We have analyzed several past online A/B tests and revealed that some of them have SRM at a statistically significant level (p<0.001). Example: we had 50%/50% traffic allocation between control and test variations. However, we ended up with the number of users below:

Control: 130631
Test: 133192

Some of the past tests had a discrepancy of 2% in the number of users between control and variation. Some had 5% or 10%.

Is there a percentage of discrepancy that can tolerate if a Sample Ratio Mismatch is detected?

Like less than 2% is tolerable, and we can still trust the results of our A/B tests. If more than 2% we can't do that.

Answer 1

The percentage of discrepancy doesn't tell you much on its own. For example:

• prop.test(c(25, 20), c(50, 50)) has a 10 percentage-point discrepancy (50% vs 40%) but the p-value is clearly non-significant ($p=0.42$)
• prop.test(c(2500, 2400), c(5000, 5000)) has a 2 percentage-point discrepancy (50% vs 48%) but the p-value is significant at the 5% level ($p=0.048$).

Therefore, what matters is the combination of the discrepancy and the sample size.

If the p-value of a SRM test is very small (e.g. $p<0.001$), that tells you that it would be extremely unlikely to get a discrepancy like the one you got (or larger) if your experimentation system was working properly. I would certainly be worried with such a small p-value.

On another note, if you are running many SRM tests on past A/B tests, you probably want to adjust for multiple comparisons. Otherwise, even if all your tests were correctly set up, you would still expect to get a $p<0.05$ approximately 5% of the time (false positive).