I am trying to optimize my Google Adwords campaign (online advertizing on Google search), and running into the following 'statistical significance' question.

I ran a campaign with many keywords (about 400 keywords), and got a total of 291 clicks (visits) and 34 conversions (clients).

Now I would like to optimize my campaign, and I am focusing on the keywords statistics. I see that a set of 10 keywords seem to perform much better than the rest:

Keyword group A (10 keywords - the "best"): 47 clicks, 16 conversions

Keyword group B (390 keywords - the rest):  244 clicks, 18 conversions

If I apply these data into a A/B Testing Significance calculator, I obtain a 99% confidence that A is better than B (I used a this free A/B testing significance calculator).

Can I consider that group A is better than group B at converting? or am I making a wrong assumption?

I am confused because all articles I have read mention A/B tests where A and B are single instances (for example just one keyword), and not group of cases (set of keywords).


1 Answer 1


Short answer: yes.

Long answer: The test you linked to is technically to compare two distributions rather than two proportions. The better test would be a chi-squared test, for example (from R):

prop.test(c(16, 47), c(18, 244))
# 2-sample test for equality of proportions with continuity correction
# data:  c(16, 47) out of c(18, 244)
# X-squared = 40.7653, df = 1, p-value = 1.717e-10

This is an academic point, since both tests show the probability is "very high" that these are in fact different groups.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.