A/B experimental design with strata or blocks? I have two approaches I wish to evaluate conversions (currently at ~2%).  As in many situations, I cannot assume equal likelihood of conversions of instances which any number of individuals can be part. There are approximately 4K possible ‘instances’ but only a handful account for the majority of the previously observed conversions.  I initially thought of setting up a form of stratified sampling to group them on to buckets to randomly assign to the two methods mentioned above to assure equal representation.
However, I have a colleague suggesting an alternative approach whereby each ‘individual’ is randomly assigned one of the approaches.  And only afterwards to measure differences whether global or clustered instances.  He says this is a form of a randomized block design.
I have been getting conflicting suggestions.  To further complicate things, there is a seasonal component to the conversion rates that I’m not sure impacts things in terms of statistics.  I’m not sure which way would result in greater statistical power.
Any advice is appreciated.
 A: With A/B testing you have to avoid situations where the decision of whether to use A or B is influenced by a factor that also influences the outcome, in your case, the conversion. Those factors are called confounders. E.g., image A is only used for clothes and B only on cars. Clothes probably have a much better conversion, so your A/B test would not work well. The confounder here is the product, which influences both the choice for A or B and the conversion rate.
The standard approach to deal with this is to properly randomize the A/B testing, i.e. make sure that the choice for A and B is really completely random, that is, not influenced by anything (like the product).
Going one step further, you can use stratified sampling to reduce the error. (IIUC, you call your strata "instances".) Here it doesn't really matter whether you first randomize and then take a weighted mean of the in-strata conversion differences, or you first stratify, then randomize in each stratum separately and take the weighted mean thereof. Just make sure that you end up with a sufficient amount of samples in each stratum.
Finally, you mentioned seasonality. You could try to account for this by including the seasonality into your stratification, e.g. use the weekday as a stratum.
