3
$\begingroup$

I'm using a service (Optimizely, as it happens), which show a confidence interval but doesn't explain what it is.

I'm assuming it is a Binomial Confidence Interval, but I have no idea what the % certainty of the range is, or the detailed meaning. Searching the Internet doesn't help - there is nobody with a clear explanation of what the default for this kind of thing is.

For example, for 97 success out of 939 trials, Optimizely says "10.33% (±1.95)".

What does that ±1.95 likely mean, exactly?

And yes, I've already complained to them that they should document it. But clearly, there is some default that statisticians habitually use. What is it?

$\endgroup$
1
  • $\begingroup$ Talking about this a bit more in the office here at ScraperWiki, it seems in some disciplines there is a different assumption. e.g. Physics uses 1 sigma (68%) only quite often. Would love good in answer to this in an answer too! $\endgroup$
    – frabcus
    Jun 24, 2014 at 14:44

3 Answers 3

3
$\begingroup$

Exactly? You want an exact answer to a statistical question? And you're willing to pay a service to provide it? Did you formerly work for AIG?

Seriously, we will need to guess here. My guess is that you are being offered an estimate of the 95% confidence interval using a Normal approximation around the point estimate of the binomial proportion for your data. My guess is informed by the fact that I have an open R console session:

> binom.test(97, 939,  10.33/100)

    Exact binomial test

data:  97 and 939
number of successes = 97, number of trials = 939, p-value = 1
alternative hypothesis: true probability of success is not equal to 0.1033
95 percent confidence interval:
 0.08457063 0.12456106
sample estimates:
probability of success 
             0.1033014 

> abs(0.08457063 - 0.12456106)
[1] 0.03999043
> abs(0.08457063 - 0.12456106)/2
[1] 0.01999522

There's an "exact" answer, but exact has a different meaning than the word you are probably asking. Furthermore, you will notice that one half the span of the 95% CI 1.99995% is not the same as your Optimizely answer of 1.95%. The exact CI's are not symmetric when the binomial proportion is far away for 0.5.

$\endgroup$
2
$\begingroup$

Following DWin's answer, I used R to find some more binomial confidence intervals:

> library(binom)
> binom.confint(97, 939)
          method  x   n      mean      lower     upper
1  agresti-coull 97 939 0.1033014 0.08535696 0.1244784
2     asymptotic 97 939 0.1033014 0.08383472 0.1227681
3          bayes 97 939 0.1037234 0.08452740 0.1233938
4        cloglog 97 939 0.1033014 0.08487358 0.1237776
5          exact 97 939 0.1033014 0.08457063 0.1245611
6          logit 97 939 0.1033014 0.08539352 0.1244537
7         probit 97 939 0.1033014 0.08514200 0.1241205
8        profile 97 939 0.1033014 0.08492041 0.1238435
9            lrt 97 939 0.1033014 0.08491887 0.1238435
10     prop.test 97 939 0.1033014 0.08493732 0.1249860
11        wilson 97 939 0.1033014 0.08542358 0.1244118

"10.33% (±1.95)" looks like the asymptotic 95% confidence interval.

$\endgroup$
1
  • $\begingroup$ Thanks for this, shows that for my purposes the calculation method doesn't matter much! $\endgroup$
    – frabcus
    Jun 24, 2014 at 14:43
1
$\begingroup$

The Optimizely documentation says:

What these numbers mean is that at the time the experiment was running, if you were to take a bunch of random samples, 95% of the time, the conversion rate would fall in the range listed. For example, if the conversion rate is 6.0% and the confidence interval is 0.11%, then 95% of the tests will have a conversion rate between 5.89% and 6.11%.

More generally, Wikipedia suggests this is common practice:

In applied practice, confidence intervals are typically stated at the 95% confidence level. However, when presented graphically, confidence intervals can be shown at several confidence levels, for example 50%, 95% and 99%.

And discussion with friends suggests it's field specific.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.