2
$\begingroup$

Recently, I took a test in statistics. The test stated something like this (I can't recall exactly the words).

You realized 5 experiments with 100000 users each to evaluate a enhancement after some interface changes. The first one (the control group) received a 10% return. The other 4 experiments had the following returns 7%, 8.5%, 12% and 14%. Is possible to draw any conclusion with 95% of confidence in any of the 4 experiments when compared with control group?

My approach to this was to make a simple z-test over a Bernoulli distribution using $\alpha = 0.05$. I used the two tailed test, with the null hypothesis as difference between control group probability and the given experiment probability is not significant. It means for example comparing the control group (10%) with the 7% experiment group that

$H_0 = p_{control} - p_{exp7} = 0$

For that I took the following approach

  1. Computed the probability of event occur

$\hat{p} = (10000 + 7000)/(100000+100000) = 0.085$

  1. Computed the standard error

$SE = \sqrt{0.085 * (1 - 0.085) * (\frac{1}{100000} + \frac{1}{100000})}$

  1. Computed the test statistics

$t = (0.1 - 0.07)/ SE \approx 24.054$

  1. Check $z_{\alpha}$ for 95% in two-sided Z-distribution $\approx 1.96$

Since the $t$ is outside the interval $[-z_{\alpha}, z_{\alpha}]$, I assume that I can reject the null-hypothesis that the difference is non-significant, hence I conclude that experiment 7% significantly worse than the control experiment.

Is that correct? Is there better tests or approaches to that? Should I use one-tailed and the null hypothesis to $\bar{X} \le \bar{Y}$? Is there any functions in Python or R that can be used to this kind of experiments, as we have for t-tests with samples**?

** In Python we can use scipy.stats.t.ttest_ind and in R we have t.test()

Improve this question
$\endgroup$
6
  • $\begingroup$ You're working too hard. What, approximately, are the standard errors of the total return in each experiment? How do the differences in total returns compare to those standard errors? $\endgroup$
    – whuber ♦
    Commented Jan 18 at 22:24
  • $\begingroup$ For each $i$-percentage we have the given the approximate $i$-th SE: ${SE}_i = \sqrt{(p * (1-p)) / (1/n)} \approx [94, 80, 88, 102, 109]$. SE total, computed with $SE_{tot} = \sqrt{\sum{SE_i^2}} \approx 214$. Computing the differences $SE_{tot} - SE_i \approx [119, 134, 126,111, 104]$, $\endgroup$
    – Lin
    Commented Jan 20 at 0:59
  • $\begingroup$ Good. And how do the differences in the total returns compare to those SEs? $\endgroup$
    – whuber ♦
    Commented Jan 20 at 17:58
  • $\begingroup$ Some are bigger the last are smaller. We can compute a ratio $\frac{SE_i}{SE_{tot} - SE_i} \approx [ 0.79, 0.60, 0.70, 0.92, 1.05 ]$. $\endgroup$
    – Lin
    Commented Jan 21 at 1:26
  • 1
    $\begingroup$ That ratio makes no sense. Your percentages, however, correspond to differences in the thousands, each of which is ten or more times any standard error: that is so large that no testing is needed. $\endgroup$
    – whuber ♦
    Commented Jan 21 at 14:33

1 Answer 1

Reset to default
2
+50
$\begingroup$

Your testing procedures look okay. The method you are using is Wald CI of binomial proportions. Some errors are

  • The null hypothesis is "the control and experiment groups have the same probability of return." Hypotheses must be regarding population parameters (i.e. probability) instead of sample estimates (i.e. proportion). Do not associate statistical significance with hypotheses. Here I am confused what return means and whether the percentages have probability interpretation. Do they actually mean percentage increases instead, so an increase from an average score of 80 to 88 leads to 10% return? If so, testing binomial proportions may not be correct.

  • The standard error of the difference should be $\text{SE} =\sqrt{p_0(1 - p_0)/n_0 + p_1(1 - p_1)/n_1}$. The formula you used assumes equal variances between two groups, which is not appropriate in most cases.

  • Using a two-sided test, the correct conclusion is that "the control and experiment groups have different probabilities of return" or "the return probability in the experiment group significantly differs from that in the control group." Again, we must talk about population parameters that are not observed but inferred.

Should I use a one-tailed test?

You can but do no have to. One-tailed tests correspond to one-sided CI (the other side is the theoretical limit, here +1 or -1 for a probability).

Is there better tests or approaches to that? Is there any functions in Python or R that can be used to this kind of experiments?

Yes, in R you should use DescTools::BinomDiffCI(x1, n1, x2, n2, conf.level = 0.95, sides = c("two.sided","left","right"), method = c("ac", "wald", "waldcc", "score", "scorecc", "mn", "mee", "blj", "ha", "hal", "jp")). As it shows, there are multiple methods to construct the confidence interval. You can check out the details by sending ??BinomDiffCI in the console and check out the script for formula by Ctrl+click BinomDiffCI() in a line in your script editor. Since your sample size is quite large, they will differ by a tiny amount.

If the task is hypothesis testing instead of confidence intervals, you can use (1) chi-square test of independence in chisq.test() (2) permutation test of independence with resampling coin::chisq_test(). Bootstrapped confidence intervals for proportions is possible but not very meaningful. See https://cran.r-project.org/web/packages/confintr/vignettes/confintr.html. In theory t.test() is not appropriate for binomial proportions although in your case it will still give very similar confidence intervals as DescTools::BinomDiffCI() does.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.