I'm a product owner responsible for running a lot of A/B tests comparing an experiment vs a control group. Usually, both groups are a subset of a larger overall population (million plus users).
I'm finding that our results can be impacted greatly by say, the average age of a user (newer the better as expected), and we've declared tests winners sometimes even when the distribution of the sample set of users is not similar to the overall population (more newer users for example).
I've been researching different statistical tests to be able to determine if the attributes of the sample users for the experiment are very similar to the attributes of the population as a whole. Because I'm not a true math or stats guy, my head is spinning. But I know bringing this level of analysis into our process will have a big impact.
Here is some data we could use to compare two groups based on last activity date, and one attribute - sends. Ideally, I would be able to have 1 or more attributes analyzed.
EXPERIMENT GROUP Activity Day Sends Day = 1 257706 Day = 2 122338 Day = 3 78609 Day = 4 57061 Day = 5 44670 Day = 6 36465 Day = 7 30546 Day = 8 26057 Day = 9 22705 Day = 10 19877 POPULATION Activity Day Sends Day = 1 8562748 Day = 2 3623711 Day = 3 2208638 Day = 4 1539338 Day = 5 1144502 Day = 6 894538 Day = 7 720023 Day = 8 548041 Day = 9 503226 Day = 10 405254
If I'm able to confidently say within a margin of error that the experiment group is distributed similarly to the population or even just the control, then I could further test if the results of the experiment vs. the control are significant and have a much more scientific way to select winners and losers. Better than simply saying "the experiment made us more money so it definitely will make us more money when we roll it out to a million users." The business users excel heavily, but I have a software background and am not opposed to using other packages for this analysis.
I've looked into paired t-tests, chi-squared, mixed ANOVA, and at this point the paired t-tests "appeared" to work if I am understanding the interpretation of p-value and rejection of null hypothesis correctly. But I'm not sure it is the right approach.
I'm Thank you so much for any tips or suggestions, I appreciate it. K_C