# How to calculate statistical significance of a difference between two groups?

I have two groups A and B. Each group involves some number of users. For example, let's imagine that the group A has 2000 unique users on Monday, while the group B has 2002 users on the same day.

For each group I measure the number of clicks that users make on a web site. For example, let's say that the users of group A made (totally) 400 clicks (the same user can make more than one click), while the users of group B made 450 clicks on the same page.

Then for each group I calculate the performance metric (in %) as follows: 100 * number of clicks / unique number of users.

PerfA  PerfB    NumUsersA   NumUsersB
20%    22%      2000        2002
21%    20%      2001        1999


For each pair of values PerfA and PerfB, I want to calculate the statistical significance of a difference.

Which statistical test should I use? If the Python/Scala library can be suggested, I would be very happy.

I found some examples of how the statistical significance is measured in Excel, but I am now sure about the statistical test (formula) that it refers to:

=NORM.S.DIST(ABS(PerfA*100-PerfB*100)/ABS(SQRT(((PerfA*100*(100-PerfB*100)/NumUsersA)+(PerfB*100*(100-Perf*100)/NumUsersB)))),1)

• Is this e.g. 20 users with a bad experience out of 2000? Or is this 20 bad events happening to a total of 2000 users (i.e. potentially more than one event per users)? Depending on this assuming e.g. a binomial or Poisson (or negative binomial) distribution would make more sense. Additionally, were people randomized to groups or did they self-select (or get assigned in some other non-random way - e.g. first 2000 users to log on versus the ones logging on later)? This would also make a huge difference (e.g. propensity scores might be needed). Apr 25, 2018 at 12:50
• @Björn: Many thanks Björn for your valuable feedback. Apparently, I had to be more clear in my question. I will add more details in a couple of minutes. Apr 25, 2018 at 12:59
• @Björn: Please check my update. I tried to be more concrete. Apr 25, 2018 at 13:10
• So one or more clicks per user possible? Random group assignment? Apr 25, 2018 at 13:30
• Do you have individual level click data? If so, you might consider a model predicting whether a user clicked anything/the number of clicks from group variables. Apr 25, 2018 at 13:34