So we're starting to do A/B testing in my company (it's a mobile app), and my CEO is pretty skeptical about the whole thing (I guess I can't blame him, especially after the results I'm seeing).
One of our first tests is to delay showing ads for 1 day vs. 2 days. The "control" in this case is 2 days, which is what we currently do. Because the first time around in our testing, we made our group sizes too small the results were very noisy with lots of outliers. So this time around, we created big groups (~8K users in each group) and I put in 2 control groups.
Here is the output from running aov in R for one of the metrics we are tracking:
Call:
aov(formula = connections ~ as.factor(group_id), data = userdata_exp104)
Residuals:
Min 1Q Median 3Q Max
-2.609 -2.262 -2.103 -0.262 113.391
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 2.6095 0.1275 20.469 < 2e-16 ***
as.factor(group_id)472 -0.5060 0.1799 -2.812 0.00493 **
as.factor(group_id)473 -0.3474 0.1798 -1.933 0.05333 .
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 6.117 on 6947 degrees of freedom
Multiple R-squared: 0.001189, Adjusted R-squared: 0.0009014
F-statistic: 4.135 on 2 and 6947 DF, p-value: 0.01604
In this case group_id 473 is actually the test group (with 1 day delay in ads) while the other two groups are the "controls". You can see that one of the controls (group_id 472) appears to be highly statistically significantly different from the other control.
I really don't know how to interpret this. In theory, the two control groups should have been the same. I don't know what to tell my CEO, other than we found no difference for the test group. Any ideas?