I have the following business request:
"Test this change on the landing page and implement it for all the visitors if the test shows improvement in the conversion rate, but if the page load speed significantly differs between the two variations, discard the experiment results and investigate"
The rationale behind it is we want to test the change, but we also want to make sure the results we are getting can only be attributed to the visual change itself, and not the change in page load speed that may occur due to poor implementation of A/B testing framework etc or some other factors.
As far as I know in our case page load speed is what they call "sanity guardrail metric".
However, I am not sure how to translate it into a proper A/B test.
What I figured out so far:
- We choose one-tail test for testing the conversion rate (right tail in our case, since we test for an improvement)
- We chose two-tailed test for the page load metric, since we test for a change in any direction.
What I have my doubts about:
- Required sample size: since we are going to monitor 2 separate metrics, one of them being a continuous metric and the other one - proportion, I am not sure how to calculate the required sample size. Intuition tells me that I should calculate the required sample size for both metrics with the specified minimum detectable effect, power and error probability and choose the larger sample size. Of course I expect the sample size for page load metric to be larger.
- Should I use some sort of correction here? Since I am going to monitor two metrics, maybe I need to apply some sort of correction (bernoully, dunett etc.), but I have doubts, since we do not seek to improve the page load speed - we only need to make sure it doesn't change, but since the page load speed metric is a subject to false positive as well, I guess something must be done about it as well. Besides, we combine one-tail metrics and two-tailed one into one test, which further complicated things.
Any help will be much appreciated, as well as any links to the relevant resources that cover similar problems.