0
$\begingroup$

I have two groups A and B. Each group involves some number of users. For example, let's imagine that the group A has 2000 unique users on Monday, while the group B has 2002 users on the same day.

For each group I measure the number of clicks that users make on a web site. For example, let's say that the users of group A made (totally) 400 clicks (the same user can make more than one click), while the users of group B made 450 clicks on the same page.

Then for each group I calculate the performance metric (in %) as follows: 100 * number of clicks / unique number of users.

PerfA  PerfB    NumUsersA   NumUsersB
20%    22%      2000        2002
21%    20%      2001        1999

For each pair of values PerfA and PerfB, I want to calculate the statistical significance of a difference.

Which statistical test should I use? If the Python/Scala library can be suggested, I would be very happy.

I found some examples of how the statistical significance is measured in Excel, but I am now sure about the statistical test (formula) that it refers to:

=NORM.S.DIST(ABS(PerfA*100-PerfB*100)/ABS(SQRT(((PerfA*100*(100-PerfB*100)/NumUsersA)+(PerfB*100*(100-Perf*100)/NumUsersB)))),1)
Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ Is this e.g. 20 users with a bad experience out of 2000? Or is this 20 bad events happening to a total of 2000 users (i.e. potentially more than one event per users)? Depending on this assuming e.g. a binomial or Poisson (or negative binomial) distribution would make more sense. Additionally, were people randomized to groups or did they self-select (or get assigned in some other non-random way - e.g. first 2000 users to log on versus the ones logging on later)? This would also make a huge difference (e.g. propensity scores might be needed). $\endgroup$
    – Björn
    Commented Apr 25, 2018 at 12:50
  • $\begingroup$ @Björn: Many thanks Björn for your valuable feedback. Apparently, I had to be more clear in my question. I will add more details in a couple of minutes. $\endgroup$
    – Markus
    Commented Apr 25, 2018 at 12:59
  • $\begingroup$ @Björn: Please check my update. I tried to be more concrete. $\endgroup$
    – Markus
    Commented Apr 25, 2018 at 13:10
  • $\begingroup$ So one or more clicks per user possible? Random group assignment? $\endgroup$
    – Björn
    Commented Apr 25, 2018 at 13:30
  • $\begingroup$ Do you have individual level click data? If so, you might consider a model predicting whether a user clicked anything/the number of clicks from group variables. $\endgroup$
    – RickyB
    Commented Apr 25, 2018 at 13:34

1 Answer 1

Reset to default
1
$\begingroup$

One logical approach would be to use a random effects Poisson model with a random user effect (reflecting that potentially results for the same user on different days might be correlated) and a fixed group effect (e.g. 0=group A and 1=group B or the other way around). The random effect could follow a normal distribution on the log-rate scale or a gamma distribution on the rate scale, whichever you can find a package for. A keyword you could google here is GLMM. You would be able to do this, as long as you have the data for each user and not just the overall numbers. If you only have the overall numbers, you might have to assume zero variability across people and use a Poisson model.

If you have a small number of days, you might want to consider a Bayesian approach, while with enough days a frequentist approach should work just fine (and presumably some standard package offers a likelihood ratio test). For the Bayesian approach, there are packages such as PyStan in Python (and similarly rstan in R).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.