# How much of a discrepency is tolerable if Sample Ratio Mismatch is detected?

We have analyzed several past online A/B tests and revealed that some of them have SRM at a statistically significant level (p<0.001). Example: we had 50%/50% traffic allocation between control and test variations. However, we ended up with the number of users below:

Control: 130631
Test: 133192


Some of the past tests had a discrepancy of 2% in the number of users between control and variation. Some had 5% or 10%.

Is there a percentage of discrepancy that can tolerate if a Sample Ratio Mismatch is detected?

Like less than 2% is tolerable, and we can still trust the results of our A/B tests. If more than 2% we can't do that.

• prop.test(c(25, 20), c(50, 50)) has a 10 percentage-point discrepancy (50% vs 40%) but the p-value is clearly non-significant ($$p=0.42$$)
• prop.test(c(2500, 2400), c(5000, 5000)) has a 2 percentage-point discrepancy (50% vs 48%) but the p-value is significant at the 5% level ($$p=0.048$$).
If the p-value of a SRM test is very small (e.g. $$p<0.001$$), that tells you that it would be extremely unlikely to get a discrepancy like the one you got (or larger) if your experimentation system was working properly. I would certainly be worried with such a small p-value.
On another note, if you are running many SRM tests on past A/B tests, you probably want to adjust for multiple comparisons. Otherwise, even if all your tests were correctly set up, you would still expect to get a $$p<0.05$$ approximately 5% of the time (false positive).