# Finding improvement of a test population

I am gathering data across two variants of my website (akin to a split test). There is an action that I log each time a user invokes it along with which variant they are included in. I also log all of the users included in my test (even if they don't perform any action).

Given a set of data for each variant now how can I tell a) What the average percentage increase/decrease is b) Whether the average percentage increase/decrease is statistically significant?

Note: This is not a binomial distribution because it isn't just about people successfully performing an action. Instead I care how many times the users in each variation perform the action.

Could I use something like eye-balling a box plot (and its IQR) to see if the two samples overlap any if at all and then just report on the percentage difference between the means?

-
Are you just trying to compare the proportion that perform the action in one variant compared to the other? if so it sounds too simple. You just compute the two proportions and model the data as at number of bernoulli trials. You would then just be comparing two binomial proportions. Am I missing something? I don't see where the boxplot you mention enters in. I don't see where there is a continuous variable being compared here. –  Michael Chernick Jul 6 '12 at 16:09
Hi Michael! It might totally be too simple but I'm still a beginner so thank you for the ideas! –  Justin Bozonier Jul 6 '12 at 16:31
whoops hit enter too soon... So the reason this isn't binomial is because users can realistically perform the same action multiple times and that matters to me. So I want to get a sense for which variant had people performing action x the most and be able to give an idea of how much more with what error rate. Does that make more sense? –  Justin Bozonier Jul 6 '12 at 16:32
When you say "which variant had people performing action x the most" do you mean that you only want the variant with the most "x's" performed, ignoring how many "people" performed the action? Just double-checking. –  Jonathan Jul 6 '12 at 17:20
Hi Jonathan! Yes. It means I'd like to know which variant, on average/median, caused users to perform action X the most, not how many people performed the action. –  Justin Bozonier Jul 6 '12 at 19:26
show 1 more comment