I am new to using stratified sampling for variance reduction on online experiments and I was hoping to get some directional insights.

I have a the result of an online experiment with following properties:

Treatment Strata Visitors Conversion Total
Variation NP 44168 931
Variation P 282879 281
Control NP 43347 1369
Control P 281445 418

Normally, from the above table, I can gather the conversion rate for control (0.55%) and variation(0.37%) and also see the observed difference against control(~ -32%). Furthermore, here on, I can also run the proportions_ztest in python to get the p value to understand whether the difference we see against control is statistically significant or not.

I now am trying to use the above strata information to see if by using the strata information whether I can detect changes against control better since the variance within strata should/expected to be smaller than the overall variance in the data which should yield to a more power to detect difference, if there is one.

The part where I am a little unsure on how to proceed is to running the significance when we have the strata information. In some of the papers, I read, they mostly refer to estimating the observable difference against control by taking the average or proportional average (in my case, since P strata is higher in proportion the difference weight from this one will ~86% and the 14% weight will come from the NP strata) of differences from each strata which I can achieve however, from here on.

Lets presume that the difference against control after looking at P and NP strata differences is 10% (meaning the conversion rate on variation is 10% higher than the control : variation CVR - Control CVR/Control CVR) how can I test to conclude whether it is statistically significant or not? I have came across a post on a difference channel where at the final step (for a continues metric) stratas were being passed into the ttest as is. However, I am a little hesitant to take this approach as it is testing one strata with respect to the other one but not necessarily to understand the win or loss against the control within strata.

Any guidance would be much appreciated

  • $\begingroup$ It's unclear what you're trying to estimate. In this example, at least, the differences in conversion rates between the strata are enormous. How would it make sense to combine them into a single difference? What would such an estimate mean? What would it represent? $\endgroup$
    – whuber
    Dec 23, 2022 at 16:34


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.