I am working for an online marketing agency and we are often testing different titles/pictures for products etc. which are then shown for example at google shopping.

If we want to test 2 variants of a title to improve the click trough rate (ctr) or conversion rate (cr), we usually randomly assigning a certain amount of products, lets say 10.000, to two different groups. The relevant KPI´ are normally quite similar, differences are being eliminated by switching products to one group or the other. In the end you end up with two groups (~ 5.000 products each) with the same KPI´s.

It will look like this:

Group   Impressions  Clicks  Orders   CTR   CR  No. Products
A       70,160       5,262    421     7.5%  8%  5042
B       74,287       5,572    446     7.5%  8%  4958

My question: is this a legit way or should this be done in a different way? The KPI´s are usually from the last 30 days, but if you use a different time period, they look already different.

What is a correct way of assigning products with different attributes to two groups? Is a random selection enough if the number is big enough?

Thank you!

  • $\begingroup$ Are you just trying to find if there is a difference in the CTR or CR between two different types of KPI's? $\endgroup$
    – a.powell
    Apr 7 '17 at 13:37
  • $\begingroup$ Yes. But would it make a difference if I would be interested in other KPI´s? $\endgroup$ Apr 7 '17 at 13:48

First and foremost you have to analyze your data to find out how it is distributed. If it is normally distributed, you could do a simple t-test which will tests if two sets of data are significantly different from one another.

Alternatively, if your data is not normal you could used something like a non-parametric Wilcoxon signed-rank test. This will compare your two KPI's if they lack normality by measuring if they came from a similar distribution.

There are plenty of other options for AB testing (a few can be found here). Another option, however, that could be intriguing since you are working with click through rates that are percentages, would be a Fischer's Exact Test. Here you can create a 2x2 contingency table:

Group   Impressions  Clicks   
A       70,160       5,262    
B       74,287       5,572 

From this you can test the null-hypothesis that the rates of A and B are the same versus the alternative hypothesis that they differ. I really like the Fischer's test for the problem you have presented. Hope this helps, and at a minimum the links should provide with more information as you decide what is best for your data and testing.

  • $\begingroup$ Thank you for your answer. I am using Beta Distribution for these tests, and for comparing the groups, a bayesian test. But my answer was more concerned with the composition of the initial groups. Do these just have to be put together randomly or should one use more sophisticated methods? $\endgroup$ Apr 7 '17 at 14:19
  • $\begingroup$ If the initial groups are from the same group they will likely not be any different from one another and I don't know why you would want to compare them. If they are different, I would simply just split them into group A and B based on the different KPI's. I don't see any need to randomly select data -- you can just leave it all in. $\endgroup$
    – a.powell
    Apr 7 '17 at 14:22
  • $\begingroup$ Well these are all different products with different prices, pictures etc. Meaning they all have a different CTR and CR. We have a wide range of products, from pants, to toys, to electronics etc. Randomly assigning them to one of two groups is how we do it now, but the two groups could end up being not comparable, due to differences in for example number of products of a certain category. Therefore the question if a random process is enough... $\endgroup$ Apr 7 '17 at 14:27
  • $\begingroup$ Like you said - the groups may not be comparable. If you're comparing toys to pants that is difficult. I don't see the need for head-to-head testing if you don't want to test the difference between two distinct products or groups. $\endgroup$
    – a.powell
    Apr 7 '17 at 14:31
  • 1
    $\begingroup$ Sorry, it must be my language, I am not very fluent in English. Since no lengthy comment discussions are wanted, i will move this to the chat. Anyway, thank you for your help! $\endgroup$ Apr 7 '17 at 14:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.