Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 3 down vote favorite
1

I am gathering data across two variants of my website (akin to a split test). There is an action that I log each time a user invokes it along with which variant they are included in. I also log all of the users included in my test (even if they don't perform any action).

Given a set of data for each variant now how can I tell a) What the average percentage increase/decrease is b) Whether the average percentage increase/decrease is statistically significant?

Note: This is not a binomial distribution because it isn't just about people successfully performing an action. Instead I care how many times the users in each variation perform the action.

Could I use something like eye-balling a box plot (and its IQR) to see if the two samples overlap any if at all and then just report on the percentage difference between the means?

share|improve this question
Are you just trying to compare the proportion that perform the action in one variant compared to the other? if so it sounds too simple. You just compute the two proportions and model the data as at number of bernoulli trials. You would then just be comparing two binomial proportions. Am I missing something? I don't see where the boxplot you mention enters in. I don't see where there is a continuous variable being compared here. – Michael Chernick Jul 6 '12 at 16:09
Hi Michael! It might totally be too simple but I'm still a beginner so thank you for the ideas! – Justin Bozonier Jul 6 '12 at 16:31
whoops hit enter too soon... So the reason this isn't binomial is because users can realistically perform the same action multiple times and that matters to me. So I want to get a sense for which variant had people performing action x the most and be able to give an idea of how much more with what error rate. Does that make more sense? – Justin Bozonier Jul 6 '12 at 16:32
When you say "which variant had people performing action x the most" do you mean that you only want the variant with the most "x's" performed, ignoring how many "people" performed the action? Just double-checking. – Jonathan Jul 6 '12 at 17:20
Hi Jonathan! Yes. It means I'd like to know which variant, on average/median, caused users to perform action X the most, not how many people performed the action. – Justin Bozonier Jul 6 '12 at 19:26
show 1 more comment

1 Answer

up vote 1 down vote

Ok Justin I understand now and think I can supply an answer. You have two distributions of counts. You can apply the Wilcoxon rank sum test to determine if the distributions differ in location. Since you do not know the direction of difference you make the test two-sided. The only problem with apply such tests to discrete data is that you can have a lot of ties in the ranks. The test works best with no ties but methods for dealing with ties can be found in good texts on nonparametrics such as Conover's Practical Nonaparametric Statistics.

http://www.amazon.com/Practical-Nonparametric-Statistics-Wiley-Probability/dp/0471160687/ref=sr_1_2?s=books&ie=UTF8&qid=1341594727&sr=1-2&keywords=William+Conover

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.