Like a paired t-test is there way to approach a paired proportion test? Here is the situation:

Say I have a Website "A" and Conversion Rate, assume 2% out of 1000 sessions, and we perform some changes on the same site and measure the site level Conversion Rate again assuming now to be 3% out of 890 sessions. If I were to conclude that my changes have made an impact statistically how should I go about it?

I know of the 2 sample independent proportion test, my only concern is the two samples I am considering here is actually not independent they have a lot of similarity, hence what should be my approach? And even if we were to use the independent test what would be the caveats or result in a miscommunication of the outcome?

  • $\begingroup$ "my only concern is the two samples I am considering here is actually not independent they have a lot of similarity" - that doesn't mean they're actually paired, you need to know who converts for Website A but not Website B or vice versa to calculate the McNemar's test statistic. $\endgroup$ – AdamO Jan 24 '19 at 20:14

If you actually had paired data, the test for pairs of proportions is called McNemar's Test. However, it appears from your description that you don't actually have paired data, because you don't know which item in the first sample is paired with an item in the second sample (if they even are the same individuals). The best you will be able to do then is a two-sample proportion test, under the independence assumption. You hope that each sample constitutes some approximation of a simple random sample of the group of users. My guess is whatever non-independence exists would just make the proportions more similar than they would be if you had managed to sample independently from one time to the other, so your error will be on the safe side (it will lower power).

To reiterate, if you DID get user information so that you could pair samples, assuming that an appreciable number were paired, you could work with the repeated users and do a McNemar's test.

Finally, if many of them were paired and many not paired (but you could tell which were paired), you'd have to visit your friendly neighborhood statistician.

| cite | improve this answer | |
  • $\begingroup$ AlaskaRon, thanks for the reply. The problem is Conversion Rate is measured by Sessions and the same session cannot be part of Pre and Post Testing, so I cannot create a Contingency Table saying Sessions which converted in Version "A" and Version "B". Though we can bring in Users, but getting information at a User Level (that they are the same users) can be a little tricky $\endgroup$ – Anurag H Oct 21 '15 at 7:43
  • $\begingroup$ See partially paired (and linked posts) for the situation with partially paired/unpaired data $\endgroup$ – kjetil b halvorsen Feb 25 '19 at 8:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.