I am gathering data across two variants of my website (akin to a split test). There is an action that I log each time a user invokes it along with which variant they are included in. I also log all of the users included in my test (even if they don't perform any action).
Given a set of data for each variant now how can I tell a) What the average percentage increase/decrease is b) Whether the average percentage increase/decrease is statistically significant?
Note: This is not a binomial distribution because it isn't just about people successfully performing an action. Instead I care how many times the users in each variation perform the action.
Could I use something like eye-balling a box plot (and its IQR) to see if the two samples overlap any if at all and then just report on the percentage difference between the means?