0
$\begingroup$

I want to run an a/b test. I'd like to check if my results in the experimental variant are extreme, under the null hypothesis. I haven't created my sample, yet, and I'd like to do a power analysis to make sure that I have a large enough sample in my experiment.

My test is whether more users convert on my new landing page than my old one.

I'm reading this paper about power analysis and sample size calculation.

an

It assumes that I know the population variance under the null hypothesis, but I have no idea how much variability there is. If I ran the experiment 100 times, I can guess it'll be normally distributed since the central limit theorem and that gets me part of the way there, I think?

How do I create the null sampling distribution so that I can calculate the power function?

Improve this question
$\endgroup$
9
  • $\begingroup$ If you don't know the variance, presumably you'll be using a t-test rather than a z-test; alternatively if you're relying on CLT + Slutsky's theorem to get to a z-test (n=100 may or may not be sufficient to do this) then you are treating the sample variance as if it were the population variance. $\endgroup$
    – Glen_b
    Commented Apr 23, 2019 at 8:49
  • $\begingroup$ Ah, I think I get it. I run a t-test to see if the statistic is different between the control and experimental groups. The idea is that there is a distribution of statistics where there is no difference, the null distribution, and my p-value determines the likelihood I observed the difference, given the null is true. $\endgroup$
    – Cauder
    Commented Apr 23, 2019 at 14:34
  • $\begingroup$ I did a datacamp class and they made the null distribution by creating permutations of the observations, which didn't make much sense to me. $\endgroup$
    – Cauder
    Commented Apr 23, 2019 at 14:34
  • 1
    $\begingroup$ 1. Getting p-values from permuting the data is perfectly sensible -- that's a permutation test. 2. The p-value is the probability of a difference (from the situation under the null) at least as large as the observed one. 3. When you say "how do I do a power analysis" ... what are you doing exactly? [To me a power analysis might be to compute a power curve (as a function of effect size) at some sample size, or to compute a power curve as a function of sample size at some effect size, but some people seem to use it to mean computing a sample size to achieve a given power at some effect size] $\endgroup$
    – Glen_b
    Commented Apr 23, 2019 at 16:11
  • 1
    $\begingroup$ Note that permutation tests are justified in a somewhat different way to the "usual" tests. For example they don't necessarily rely on random sampling of the population; they can rely on random assignment to treatment even without being a random sample of the population and show a significant effect (the difficulty may be in arguing that the result extends to the broader population). $\endgroup$
    – Glen_b
    Commented Apr 23, 2019 at 16:15

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.