I want to run an a/b test. I'd like to check if my results in the experimental variant are extreme, under the null hypothesis. I haven't created my sample, yet, and I'd like to do a power analysis to make sure that I have a large enough sample in my experiment.
My test is whether more users convert on my new landing page than my old one.
I'm reading this paper about power analysis and sample size calculation.
It assumes that I know the population variance under the null hypothesis, but I have no idea how much variability there is. If I ran the experiment 100 times, I can guess it'll be normally distributed since the central limit theorem and that gets me part of the way there, I think?
How do I create the null sampling distribution so that I can calculate the power function?