I'm running a test where two groups of people are shown two different sets of search results or suggestions. We track what percentage of results each person is happy with. I've shown histograms for this below (percent approval on x axis and num people on y axis)

What statistical tests can I use to determine the statistical significance of my result that one algorithm is better than the other?

enter image description here


So the statistical question would be "does the average approval percentage rate differ between the two groups?"

The hypothesis can be answered using a student's t test, which will give you the estimate of the difference of the means between the two groups and also the variance of this estimate.

It can be used to produce a confidence interval of the difference and if this confidence interval does not contain zero, then you can say that they do statistically differ.

  • $\begingroup$ check here for the details on the calculation and procedure: en.wikipedia.org/wiki/… ; In R this can be called with t.test $\endgroup$ – cole Jun 18 '15 at 17:49
  • 1
    $\begingroup$ I thought that a t test could only be used for normally distributed data? The histogram looks something more like an exponential or gamma distribution, so I didn't think a t test would apply here. $\endgroup$ – Ben McCann Jun 18 '15 at 18:19
  • $\begingroup$ @BenMcCann I'm assuming that your sample sizes are fairly large if this is from an A/B web campaign. In that case, normality is not required for the assumption of a t-test. If you are still worried about that however, you can use a Wilcoxon signed-rank test: en.wikipedia.org/wiki/Wilcoxon_signed-rank_test This is just about as good as a t-test for normal populations and definitely better when using non-normal populations. $\endgroup$ – cole Jun 19 '15 at 13:35

Let the measurements for the active (experiment) group be $p^A_1\ldots,p^A_m$, and the measurements for the control group be $p^C_1,\ldots,p^C_n$. Put $E(p_i^A) = \mu_A$, and $Var(p_i^A) = \sigma_A^2$ for all $i$ (so I'm assuming the $p_i^A$s are iid). Similarly, put and $E(p_i^C) = \mu_C$ and $Var(p_i^C) = \sigma_C^2$, for all $i$. Assume that $p_i^A$s and $p_j^C$s are all independent of each other.

Your goal is to test whether $\mu_A = \mu_C$ or $\mu_A \neq \mu_C$. Let $\bar{X}^A = \sum_1^m p_i^A/m$ and $\bar{X}^C = \sum_1^n p_i^C/n$. Then $\bar{X}^A - \bar{X}^C$ is an unbiased estimator of $\mu_A - \mu_C$. Put $N = m + n$. Let $S_A^2$ and $S_C^2$ be sample variances for the active and the control groups respectively.

If $m$, $n$, and $N$ are such that $m/N \to \rho$ and $n/N \to 1-\rho$ as $m,n \to \infty$, with $0 < \rho <1$, you can show that the CLT implies as $N\to\infty$

\begin{equation} \sqrt{N}\frac{\left[(\bar{X}^A -\bar{X}^C) - (\mu_A - \mu_C)\right]}{\sqrt{\frac{S_A^2}{m} + \frac{S_C^2}{n}}} \xrightarrow{D} N(0,1). \end{equation}

From here, we can get an approximate $(1-\alpha)\%$ confidence interval for $\mu_A - \mu_C$. Let $z_{\alpha/2}$ be the unique positive real number such that $1- \Phi(z_{\alpha/2}) = \alpha/2$, where $\Phi$ is the CDF of the standard normal distribution. Using the above convergence result, it is not hard to see that the following is an approximate $(1-\alpha)\%$ confidence interval for $\mu_A - \mu_C$:

\begin{equation} \bar{x}^A - \bar{x}^C \pm z_{\alpha/2}\sqrt{\frac{s_A^2}{m} + \frac{s_C^2}{n}}, \end{equation} where $\bar{x}^A$, $\bar{x}^C$, $s_A^2$, and $s_C^2$ are the realized values of sample mean and the sample variance. Now, if 0 doesn't belongs to this confidence interval, we can reject the hypothesis that $\mu_A = \mu_C$ at level $1 - \alpha$.

To summarize, you don't need normality of your underlying distributions as long as your active and control groups are large enough and the above formula gives you a method to calculate an approximate confidence interval for $\mu_A - \mu_C$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.