2
$\begingroup$

I am running a test in Google Analytics to find the impact of a new functionality release for sharing stuff.

The setup follows:

  • There is a control group which continues to see and use the old "sharing" functionality
  • There is an exposed group which sees and uses the new "sharing" functionality"

I am trying to identify whether the new functionality performs better.

I believe I am trying to answer 2 questions:

  1. Do more people share with the new functionality?
  2. Out of those who do share, has the average quantity of shares increased?

For the first question I have run a Chi-Squared test and got the following results: http://www.evanmiller.org/ab-testing/chi-squared.html#!1851/21926;46/926@95

Basically there were 21926 users in control and 1851 shared something (one or more shares), and there were 926 users exposed and 46 shared something (one or more shares).

My question are:

  1. Am I setting up this test up correctly and using the right statistical methodology?
  2. What statistical methodology should I use to measure the result for question 2? Assuming that for the control group the 1851 users shared 4321 items, and for the exposed group the 46 users should 133 items.
$\endgroup$
2
  • $\begingroup$ I think you can use t-test (if the assumptions are met) for both objectives. $\endgroup$ – Dr Nisha Arora Mar 21 '16 at 11:31
  • $\begingroup$ This is a good start, but it is not set up correctly. Instead of having the # of trials, you need the # of FAILS which is trials-successes. $\endgroup$ – Toni Rosati Jan 9 '17 at 18:24
1
$\begingroup$

This is a good start, but the presentation could be better. Present confidence intervals for each proportion, and for their difference. For comparison, here is some of the calculations in R:

 tab <- matrix(c(1851, 21926-1851, 46, 926-46), 2, 2, byrow=TRUE)
chisq.test(tab)

    Pearson's Chi-squared test with Yates' continuity correction

data:  tab
X-squared = 13.637, df = 1, p-value = 0.0002218

 prop.test(tab)

    2-sample test for equality of proportions with continuity correction

data:  tab
X-squared = 13.637, df = 1, p-value = 0.0002218
alternative hypothesis: two.sided
95 percent confidence interval:
 0.01971147 0.04977713
sample estimates:
    prop 1     prop 2 
0.08442032 0.04967603 

For the group-wise proportions confidence intervals, you need binomial confidence intervals: (don't look at the p-values which are for an irrelevant test, only the confidence interval)

 binom.test(tab[1, ])

    Exact binomial test

data:  tab[1, ]
number of successes = 1851, number of trials = 21926, p-value < 2.2e-16
alternative hypothesis: true probability of success is not equal to 0.5
95 percent confidence interval:
 0.08077285 0.08817840
sample estimates:
probability of success 
            0.08442032 

 binom.test(tab[2, ])

    Exact binomial test

data:  tab[2, ]
number of successes = 46, number of trials = 926, p-value < 2.2e-16
alternative hypothesis: true probability of success is not equal to 0.5
95 percent confidence interval:
 0.0365953 0.0657078
sample estimates:
probability of success 
            0.04967603 

Then your question 2:
This can be modeled via an Poisson rate regression (search this site), but in this case it is simpler to use the poisson.test function which compares two Poisson rates:

 poisson.test(c(control=4321, exposed=133), 
+              c(control=1851, exposed=133))
> 

    Comparison of Poisson rates

data:  c(control = 4321, exposed = 133) time base: c(control = 1851, exposed = 133)
count1.control = 4321, expected count1.control = 4155.4, p-value <
2.2e-16
alternative hypothesis: true rate ratio is not equal to 1
95 percent confidence interval:
 1.964217 2.795353
sample estimates:
rate ratio.control 
          2.334414 
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.