0
$\begingroup$

I'm trying to compare the values of two counts from an A/B test, $c_1$ and $c_2$.

If I assume the data is Poisson distributed then the mean and variance are $\lambda$ and the standard error is $\frac {\lambda}{n}$. Since we only have one observation for each count the means and SE are just $c_1$ and $c_2$ for each. If $\lambda$ is sufficiently large, I can assume that the Poisson distribution approximates a normal distribution. I'm therefore creating a test statistic which is $\frac{c2-c1}{\sqrt{c1+c2}}$, and comparing this to 1.96 to check the significance of the difference.

Is this a valid procedure?

$\endgroup$
1
  • $\begingroup$ This sounds reasonable so long as lambda is large enough for the approximation to be good. A quick simulation in R shows that the test has the appropriate false positive rate. $\endgroup$ – Demetri Pananos Jul 29 '19 at 2:05
1
$\begingroup$

The generally accepted way of comparing 2 samples from Poisson distributions, to see if they have the same mean, is to note that the sum of 2 Poisson random variables $X_1$ and $X_2$ with means $\lambda_1$ and $\lambda_2$ is Poisson with mean $\lambda_1 + \lambda_2$; and the conditional distribution of $X_1$ given $X_1 + X_2 = n$ is $Binomial(n, \lambda_1 / (\lambda_1 + \lambda_2))$.

So you would compute the probability of observing $X_2 = c_2$ under that.

See http://www.biostat.jhsph.edu/~bcaffo/651_2006/files/lecture27.pdf

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.