Group composition for hypothesis testing I am working for an online marketing agency and we are often testing different titles/pictures for products etc. which are then shown for example at google shopping. 
If we want to test 2 variants of a title to improve the click trough rate (ctr) or conversion rate (cr), we usually randomly assigning a certain amount of products, lets say 10.000, to two different groups. The relevant KPI´ are normally quite similar, differences are being eliminated by switching products to one group or the other. In the end you end up with two groups (~ 5.000 products each) with the same KPI´s. 
It will look like this:
Group   Impressions  Clicks  Orders   CTR   CR  No. Products
A       70,160       5,262    421     7.5%  8%  5042
B       74,287       5,572    446     7.5%  8%  4958

My question: is this a legit way or should this be done in a different way? 
The KPI´s are usually from the last 30 days, but if you use a different time period, they look already different. 
What is a correct way of assigning products with different attributes to two groups? Is a random selection enough if the number is big enough? 
Thank you!
 A: First and foremost you have to analyze your data to find out how it is distributed. If it is normally distributed, you could do a simple t-test which will tests if two sets of data are significantly different from one another.
Alternatively, if your data is not normal you could used something like a non-parametric Wilcoxon signed-rank test. This will compare your two KPI's if they lack normality by measuring if they came from a similar distribution.
There are plenty of other options for AB testing (a few can be found here). Another option, however, that could be intriguing since you are working with click through rates that are percentages, would be a Fischer's Exact Test. Here you can create a 2x2 contingency table:
Group   Impressions  Clicks   
A       70,160       5,262    
B       74,287       5,572 

From this you can test the null-hypothesis that the rates of A and B are the same versus the alternative hypothesis that they differ. I really like the Fischer's test for the problem you have presented. Hope this helps, and at a minimum the links should provide with more information as you decide what is best for your data and testing. 
