I am running a test in Google Analytics to find the impact of a new functionality release for sharing stuff.
The setup follows:
- There is a control group which continues to see and use the old "sharing" functionality
- There is an exposed group which sees and uses the new "sharing" functionality"
I am trying to identify whether the new functionality performs better.
I believe I am trying to answer 2 questions:
- Do more people share with the new functionality?
- Out of those who do share, has the average quantity of shares increased?
For the first question I have run a Chi-Squared test and got the following results: http://www.evanmiller.org/ab-testing/chi-squared.html#!1851/21926;46/926@95
Basically there were 21926 users in control and 1851 shared something (one or more shares), and there were 926 users exposed and 46 shared something (one or more shares).
My question are:
- Am I setting up this test up correctly and using the right statistical methodology?
- What statistical methodology should I use to measure the result for question 2? Assuming that for the control group the 1851 users shared 4321 items, and for the exposed group the 46 users should 133 items.