Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Like a paired t-test is there way to approach a paired proportion test? Here is the situation:
Say I have a Website "A" and Conversion Rate, assume 2% out of 1000 sessions, and we perform some changes on the same site and measure the site level Conversion Rate again assuming now to be 3% out of 890 sessions. If I were to conclude that my changes have made an impact statistically how should I go about it?
If you actually had paired data, the test for pairs of proportions is called McNemar's Test. However, it appears from your description that you don't actually have paired data, because you don't know which item in the first sample is paired with an item in the second sample (if they even are the same individuals). The best you will be able to do then is a two-sample proportion test, under the independence assumption. You hope that each sample constitutes some approximation of a simple random sample of the group of users. My guess is whatever non-independence exists would just make the proportions more similar than they would be if you had managed to sample independently from one time to the other, so your error will be on the safe side (it will lower power).
To reiterate, if you DID get user information so that you could pair samples, assuming that an appreciable number were paired, you could work with the repeated users and do a McNemar's test.
Finally, if many of them were paired and many not paired (but you could tell which were paired), you'd have to visit your friendly neighborhood statistician.
