I am new to using stratified sampling for variance reduction on online experiments and I was hoping to get some directional insights.
I have a the result of an online experiment with following properties:
| Treatment | Strata | Visitors | Conversion | Total |
|---|---|---|---|---|
| Variation | NP | 44168 | 931 | |
| Variation | P | 282879 | 281 | |
| Control | NP | 43347 | 1369 | |
| Control | P | 281445 | 418 |
Normally, from the above table, I can gather the conversion rate for control (0.55%) and variation(0.37%) and also see the observed difference against control(~ -32%). Furthermore, here on, I can also run the proportions_ztest in python to get the p value to understand whether the difference we see against control is statistically significant or not.
I now am trying to use the above strata information to see if by using the strata information whether I can detect changes against control better since the variance within strata should/expected to be smaller than the overall variance in the data which should yield to a more power to detect difference, if there is one.
The part where I am a little unsure on how to proceed is to running the significance when we have the strata information. In some of the papers, I read, they mostly refer to estimating the observable difference against control by taking the average or proportional average (in my case, since P strata is higher in proportion the difference weight from this one will ~86% and the 14% weight will come from the NP strata) of differences from each strata which I can achieve however, from here on.
Lets presume that the difference against control after looking at P and NP strata differences is 10% (meaning the conversion rate on variation is 10% higher than the control : variation CVR - Control CVR/Control CVR) how can I test to conclude whether it is statistically significant or not? I have came across a post on a difference channel where at the final step (for a continues metric) stratas were being passed into the ttest as is. However, I am a little hesitant to take this approach as it is testing one strata with respect to the other one but not necessarily to understand the win or loss against the control within strata.
Any guidance would be much appreciated
