I am working for an online marketing agency and we are often testing different titles/pictures for products etc. which are then shown for example at google shopping.
If we want to test 2 variants of a title to improve the click trough rate (ctr) or conversion rate (cr), we usually randomly assigning a certain amount of products, lets say 10.000, to two different groups. The relevant KPI´ are normally quite similar, differences are being eliminated by switching products to one group or the other. In the end you end up with two groups (~ 5.000 products each) with the same KPI´s.
It will look like this:
Group Impressions Clicks Orders CTR CR No. Products A 70,160 5,262 421 7.5% 8% 5042 B 74,287 5,572 446 7.5% 8% 4958
My question: is this a legit way or should this be done in a different way? The KPI´s are usually from the last 30 days, but if you use a different time period, they look already different.
What is a correct way of assigning products with different attributes to two groups? Is a random selection enough if the number is big enough?