1
$\begingroup$

So we're starting to do A/B testing in my company (it's a mobile app), and my CEO is pretty skeptical about the whole thing (I guess I can't blame him, especially after the results I'm seeing).

One of our first tests is to delay showing ads for 1 day vs. 2 days. The "control" in this case is 2 days, which is what we currently do. Because the first time around in our testing, we made our group sizes too small the results were very noisy with lots of outliers. So this time around, we created big groups (~8K users in each group) and I put in 2 control groups.

Here is the output from running aov in R for one of the metrics we are tracking:

Call:
aov(formula = connections ~ as.factor(group_id), data = userdata_exp104)

Residuals:
    Min      1Q  Median      3Q     Max 
 -2.609  -2.262  -2.103  -0.262 113.391 

Coefficients:
                       Estimate Std. Error t value Pr(>|t|)    
(Intercept)              2.6095     0.1275  20.469  < 2e-16 ***
as.factor(group_id)472  -0.5060     0.1799  -2.812  0.00493 ** 
as.factor(group_id)473  -0.3474     0.1798  -1.933  0.05333 .  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 6.117 on 6947 degrees of freedom
Multiple R-squared:  0.001189,  Adjusted R-squared:  0.0009014 
F-statistic: 4.135 on 2 and 6947 DF,  p-value: 0.01604

In this case group_id 473 is actually the test group (with 1 day delay in ads) while the other two groups are the "controls". You can see that one of the controls (group_id 472) appears to be highly statistically significantly different from the other control.

I really don't know how to interpret this. In theory, the two control groups should have been the same. I don't know what to tell my CEO, other than we found no difference for the test group. Any ideas?

Improve this question
$\endgroup$
2
  • $\begingroup$ Is the response variable "connections" a binary outcome (1 if connection occurs and 0 if not)? $\endgroup$
    – Deathkill14
    May 22, 2014 at 17:41
  • $\begingroup$ No, it is an integer variable (i.e. 1, 2, 3, etc). I should say it is strictly positive and many users have 0, so the distribution appears heavily left-skewed. $\endgroup$
    – thecity2
    May 22, 2014 at 19:56

2 Answers 2

Reset to default
1
$\begingroup$

ANOVA (analysis of variance) is an omnibus test, meaning it jointly tests a bunch of null hypotheses of the form $\mu_{1} = \mu_{2} = \dots = \mu_{k}$. If you reject the null hypothesis, all you know is that at least one group mean differs from at least one other group mean at your chosen significance level. In order to narrow down which means are different from which, you need to perform post hoc pairwise tests (i.e. t tests). Doing these post hoc tests alters the meaning of $\alpha$ (your significance level), especially when the number of groups ($k$) is large, and you should give some thought to methods to control the family-wise error rate, or to control the false discovery rate.

While the ANOVA is generally appropriate only for (approximately) normally-distributed variables with equal variances, there are a variety of nonparametric omnibus and post hoc tests that permit inference if your data do not meet these requirements.

Improve this answer
$\endgroup$
0
$\begingroup$

Use contrasts with your anova/regression models to test specific hypothesis. I would suggest to use the helmert contrasts or effects coding to test out hypothesis of whether there are any differences between specific groups within the factor.

use either :

aov(formula = connections ~ as.factor(group_id), data = userdata_exp104, contrasts =list(as.factor(group_id)="contr.helmert"))

or

aov(formula = connections ~ as.factor(group_id), data = userdata_exp104, contrasts =list(as.factor(group_id)="contr.sum"))

Note that the coefficients have a different meaning here when you use contrasts compared to the normal anova model. The interpretation of your results depend on the contrast being used.

Improve this answer
$\endgroup$
6
  • $\begingroup$ So in this case I know that the two control groups should not be different, so does that mean I should just report that I found no "meaningful" differences or something to that effect? $\endgroup$
    – thecity2
    May 22, 2014 at 18:09
  • $\begingroup$ According to your anova model your intercept holds the value of the average of all levels + the control level(not the 472). So the coefficients you see here are the difference of that level against that control level. Infact it says that your two control levels are very different, more different than that between the control and test $\endgroup$
    – Abhilash
    May 22, 2014 at 18:14
  • $\begingroup$ Right, that's my entire question. I know for a fact the two control groups did not have any "treatment". So I'm trying to figure out how to explain the seemingly unusual result. $\endgroup$
    – thecity2
    May 22, 2014 at 19:35
  • $\begingroup$ What is different among the two control variables? $\endgroup$
    – Abhilash
    May 22, 2014 at 19:39
  • $\begingroup$ What is different among the two control groups selected ??Differences could arise out of multiple reasons, some confounding factor is lurking which you have not considered. Or your sample selection could have been biased. Or the assumptions of normality which ANOVA uses have been violated (Using a large sample does to an extent take care of this alone) $\endgroup$
    – Abhilash
    May 22, 2014 at 19:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.