Not sure how to interpret "positive" results when using more than one control group So we're starting to do A/B testing in my company (it's a mobile app), and my CEO is pretty skeptical about the whole thing (I guess I can't blame him, especially after the results I'm seeing).
One of our first tests is to delay showing ads for 1 day vs. 2 days. The "control" in this case is 2 days, which is what we currently do. Because the first time around in our testing, we made our group sizes too small the results were very noisy with lots of outliers. So this time around, we created big groups (~8K users in each group) and I put in 2 control groups. 
Here is the output from running aov in R for one of the metrics we are tracking:
Call:
aov(formula = connections ~ as.factor(group_id), data = userdata_exp104)

Residuals:
    Min      1Q  Median      3Q     Max 
 -2.609  -2.262  -2.103  -0.262 113.391 

Coefficients:
                       Estimate Std. Error t value Pr(>|t|)    
(Intercept)              2.6095     0.1275  20.469  < 2e-16 ***
as.factor(group_id)472  -0.5060     0.1799  -2.812  0.00493 ** 
as.factor(group_id)473  -0.3474     0.1798  -1.933  0.05333 .  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 6.117 on 6947 degrees of freedom
Multiple R-squared:  0.001189,  Adjusted R-squared:  0.0009014 
F-statistic: 4.135 on 2 and 6947 DF,  p-value: 0.01604

In this case group_id 473 is actually the test group (with 1 day delay in ads) while the other two groups are the "controls". You can see that one of the controls (group_id 472) appears to be highly statistically significantly different from the other control.
I really don't know how to interpret this. In theory, the two control groups should have been the same. I don't know what to tell my CEO, other than we found no difference for the test group. Any ideas?
 A: ANOVA (analysis of variance) is an omnibus test, meaning it jointly tests a bunch of null hypotheses of the form $\mu_{1} = \mu_{2} = \dots = \mu_{k}$. If you reject the null hypothesis, all you know is that at least one group mean differs from at least one other group mean at your chosen significance level. In order to narrow down which means are different from which, you need to perform post hoc pairwise tests (i.e. t tests). Doing these post hoc tests alters the meaning of $\alpha$ (your significance level), especially when the number of groups ($k$) is large, and you should give some thought to methods to control the family-wise error rate, or to control the false discovery rate.
While the ANOVA is generally appropriate only for (approximately) normally-distributed variables with equal variances, there are a variety of nonparametric omnibus and post hoc tests that permit inference if your data do not meet these requirements.
A: Use contrasts with your anova/regression models to test specific hypothesis. I would suggest to use the helmert contrasts or effects coding to test out hypothesis of whether there are any differences between specific groups within the factor.
use either :
aov(formula = connections ~ as.factor(group_id), data = userdata_exp104, contrasts =list(as.factor(group_id)="contr.helmert"))
or 
aov(formula = connections ~ as.factor(group_id), data = userdata_exp104, contrasts =list(as.factor(group_id)="contr.sum"))
Note that the coefficients have a different meaning here when you use contrasts compared to the normal anova model. The interpretation of your results depend on the contrast being used.
