Statistical significance of results in conversion data on group of Adwords keywords

Question

I am trying to optimize my Google Adwords campaign (online advertizing on Google search), and running into the following 'statistical significance' question.

I ran a campaign with many keywords (about 400 keywords), and got a total of 291 clicks (visits) and 34 conversions (clients).

Now I would like to optimize my campaign, and I am focusing on the keywords statistics. I see that a set of 10 keywords seem to perform much better than the rest:

Keyword group A (10 keywords - the "best"): 47 clicks, 16 conversions

Keyword group B (390 keywords - the rest):  244 clicks, 18 conversions

If I apply these data into a A/B Testing Significance calculator, I obtain a 99% confidence that A is better than B (I used a this free A/B testing significance calculator).

Can I consider that group A is better than group B at converting? or am I making a wrong assumption?

I am confused because all articles I have read mention A/B tests where A and B are single instances (for example just one keyword), and not group of cases (set of keywords).

Answer 1

Short answer: yes.

Long answer: The test you linked to is technically to compare two distributions rather than two proportions. The better test would be a chi-squared test, for example (from R):

prop.test(c(16, 47), c(18, 244))
# 2-sample test for equality of proportions with continuity correction
# 
# data:  c(16, 47) out of c(18, 244)
# X-squared = 40.7653, df = 1, p-value = 1.717e-10

This is an academic point, since both tests show the probability is "very high" that these are in fact different groups.