I'm a product owner responsible for running a lot of A/B tests comparing an experiment vs a control group. Usually, both groups are a subset of a larger overall population (million plus users).

I'm finding that our results can be impacted greatly by say, the average age of a user (newer the better as expected), and we've declared tests winners sometimes even when the distribution of the sample set of users is not similar to the overall population (more newer users for example).

I've been researching different statistical tests to be able to determine if the attributes of the sample users for the experiment are very similar to the attributes of the population as a whole. Because I'm not a true math or stats guy, my head is spinning. But I know bringing this level of analysis into our process will have a big impact.

Here is some data we could use to compare two groups based on last activity date, and one attribute - sends. Ideally, I would be able to have 1 or more attributes analyzed.

Activity Day    Sends
Day = 1 257706
Day = 2 122338
Day = 3 78609
Day = 4 57061
Day = 5 44670
Day = 6 36465
Day = 7 30546
Day = 8 26057
Day = 9 22705
Day = 10    19877

Activity Day    Sends
Day = 1 8562748
Day = 2 3623711
Day = 3 2208638
Day = 4 1539338
Day = 5 1144502
Day = 6 894538
Day = 7 720023
Day = 8 548041
Day = 9 503226
Day = 10    405254

If I'm able to confidently say within a margin of error that the experiment group is distributed similarly to the population or even just the control, then I could further test if the results of the experiment vs. the control are significant and have a much more scientific way to select winners and losers. Better than simply saying "the experiment made us more money so it definitely will make us more money when we roll it out to a million users." The business users excel heavily, but I have a software background and am not opposed to using other packages for this analysis.

I've looked into paired t-tests, chi-squared, mixed ANOVA, and at this point the paired t-tests "appeared" to work if I am understanding the interpretation of p-value and rejection of null hypothesis correctly. But I'm not sure it is the right approach.

I'm Thank you so much for any tips or suggestions, I appreciate it. K_C


Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.