12
$\begingroup$

Consider an experiment that outputs a ratio $X_i$ between 0 and 1. How this ratio is obtained should not be relevant in this context. It was elaborated in a previous version of this question, but removed for clarity after a discussion on meta.

This experiment is repeated $n$ times, while $n$ is small (about 3-10). The $X_i$ are assumed to be independent and identically distributed. From these we estimate the mean by calculating the average $\overline X$, but how to calculate a corresponding confidence interval $[U,V]$?

When using the standard approach for calculating confidence intervals, $V$ is sometimes larger than 1. However, my intuition is that the correct confidence interval...

  1. ... should be within the range 0 and 1
  2. ... should get smaller with increasing $n$
  3. ... is roughly in the order of the one calculated using the standard approach
  4. ... is calculated by a mathematically sound method

These are not absolute requirements, but I would at least like to understand why my intuition is wrong.

Calculations based on existing answers

In the following, the confidence intervals resulting from the existing answers are compared for $\{X_i\} = \{0.985,0.986,0.935,0.890,0.999\}$.

Standard Approach (aka "School Math")

$\overline X = 0.959$, $\sigma^2 = 0.0204$, thus the 99% confidence interval is $[0.865,1.053]$. This contradicts intuition 1.

Cropping (suggested by @soakley in the comments)

Just using the standard approach then providing $[0.865,1.000]$ as result is easy to do. But are we allowed to do that? I am not yet convinced that the lower boundary just stays constant (--> 4.)

Logistic Regression Model (suggested by @Rose Hartman)

Transformed data: $\{4.18,4.25,2.09,2.66,6.90\}$ Resulting in $[0.173,7.87]$, transforming it back results in $[0.543,0.999]$. Obviously, the 6.90 is an outlier for the transformed data while the 0.99 is not for the untransformed data, resulting in a confidence interval that is very large. (--> 3.)

Binomial proportion confidence interval (suggested by @Tim)

The approach looks quite good, but unfortunately it does not fit the experiment. Just combining the results and interpreting it as one large repeated Bernoulli experiment as suggested by @ZahavaKor results in the following:

$985+986+890+935+999 = 4795$ out of $5*1000$ in total. Feeding this into the Adj. Wald calculator gives $[0.9511,0.9657]$. This does not seem to be realistic, because not a single $X_i$ is inside that interval! (--> 3.)

Bootstrapping (suggested by @soakley)

With $n=5$ we have 3125 possible permutations. Taking the $\frac{3093}{3125} = 0.99$ middle means of the permutations, we get $[0.91,0.99]$. Looks not that bad, though I would expect a larger interval (--> 3.). However, it is per construction never larger than $[min(X_i),max(X_i)]$. Thus for a small sample it will rather grow than shrink for increasing $n$ (--> 2.). This is at least what happens with the samples given above.

$\endgroup$
  • $\begingroup$ You are correct in your second approach. I am not sure about the first one - it's not stated clearly in statistical terms. As far as I know, reproducibility means that the same experiment is performed by a different researcher and they get similar results. You need to specify your goal more clearly, preferably in terms of a statistical hypothesis regarding the parameter that you are trying to estimate. Just using the term "reproducibility" is too vague in my opinion. $\endgroup$ – Zahava Kor Feb 23 '17 at 17:49
  • $\begingroup$ You are right, repeatability is the correct term and not reproducibility. I will try to construct a definition in statistical terms. $\endgroup$ – koalo Feb 23 '17 at 18:14
  • $\begingroup$ @ZahavaKor I removed my underspecified example about repeatability and specified my actual application hoping that it clarifies my issue and not confuses. $\endgroup$ – koalo Feb 23 '17 at 20:46
  • $\begingroup$ If you are truly taking samples of size 1000, then you have not correctly applied the resampling approach. But with that much data, you don't need resampling and should get good results (that is, narrow confidence intervals) with the standard binomial approach, as you found above. Just because your individual data points aren't in the resulting interval doesn't mean the interval is incorrect. $\endgroup$ – soakley Feb 24 '17 at 19:00
  • 1
    $\begingroup$ Well, think about this. You sample 10 items and get 9 successes. I sample 1000 and get 900 successes. Who will have the more accurate estimate of the mean? Try using the formula referenced by Tim if the intuition isn't there yet. So in the last example in your question, the sample size is not 5, it is 5000! $\endgroup$ – soakley Feb 24 '17 at 19:39
6
$\begingroup$

First, to clarify, what you're dealing with is not quite a binomial distribution, as your question suggests (you refer to it as a Bernoulli experiment). Binomial distributions are discrete --- the outcome is either success or failure. Your outcome is a ratio each time you run your experiment, not a set of successes and failures that you then calculate one summary ratio on. Because of that, methods for calculating a binomial proportion confidence interval will throw away a lot of your information. And yet you're correct that it's problematic to treat this as though it's normally distributed since you can get a CI that extends past the possible range of your variable.

I recommend thinking about this in terms of logistic regression. Run a logistic regression model with your ratio variable as the outcome and no predictors. The intercept and its CI will give you what you need in logits, and then you can convert it back to proportions. You can also just do the logistic conversion yourself, calculate the CI and then convert back to the original scale. My python is terrible, but here's how you could do that in R:

set.seed(24601)
data <- rbeta(100, 10, 3)
hist(data)

histogram of raw data

data_logits <- log(data/(1-data)) 
hist(data_logits)

histogram of logit transformed data

# calculate CI for the transformed data
mean_logits <- mean(data_logits)
sd <- sd(data_logits)
n <- length(data_logits)
crit_t99 <- qt(.995, df = n-1) # for a CI99
ci_lo_logits <- mean_logits - crit_t * sd/sqrt(n)
ci_hi_logits <- mean_logits + crit_t * sd/sqrt(n)

# convert back to ratio
mean <- exp(mean_logits)/(1 + exp(mean_logits))
ci_lo <- exp(ci_lo_logits)/(1 + exp(ci_lo_logits))
ci_hi <- exp(ci_hi_logits)/(1 + exp(ci_hi_logits))

Here are the lower and upper bounds on a 99% CI for these data:

> ci_lo
[1] 0.7738327
> ci_hi
[1] 0.8207924
$\endgroup$
  • $\begingroup$ That sounds like a good approach, however the results are not what I would expect intuitively: The data_logits for 0.99,0.94,0.94 is 4.59,2.75,2,75, giving a confidence interval of [-2.73,9.47]. Transforming this back gives [0.061,0.999] - much larger than I would expect. $\endgroup$ – koalo Feb 23 '17 at 8:20
  • 1
    $\begingroup$ For just three observations, you should expect a very large confidence interval. From your histogram, it looks like you have many more than three observations --- I assumed your example with 0.99,0.94,0.94 was just to illustrate. If your actual sample size is three, I don't recommend calculating confidence intervals at all (or means, for that matter). $\endgroup$ – Rose Hartman Feb 23 '17 at 15:34
  • $\begingroup$ The histogram above comes from the python script to illustrate my problem. I am not able to get that many measurements from the real-world experiment. At least not for every combination of parameters. I agree that 3 might be too small and maybe about 10 will be possible in the final evaluation, but certainly not much more. So what should I do about that to demonstrate that I was not just lucky to get a single measurement, but that repeating the experiment does not give completely different results? $\endgroup$ – koalo Feb 23 '17 at 17:14
  • $\begingroup$ @RoseHartman That's a nice clear description but it would also be nice to see your method applied to the sample of data (n=5) in the question. $\endgroup$ – PM. Mar 2 '17 at 8:57
  • $\begingroup$ @scitamehtam I wrote my answer before koalo provided the example data and clarified that the sample size would be 10 or fewer observations. koalo has since updated the original question to include worked examples from each answer method with the n=5 data, very helpfully. $\endgroup$ – Rose Hartman Mar 2 '17 at 9:01
3
$\begingroup$

You might want to try resampling/bootstrapping. Let's look at the simple case you mentioned.

With 3 data points of 0.99, 0.94, and 0.94, you wouldn't even do the resampling because you can just list out all 27 possible permutations, find the mean in each case, and then sort the means.

If you create the list and take the middle 25 observations, you have a $25/27=$ 92.6% confidence interval of [0.9400, 0.9733]. If you want to increase the confidence to $26/27=$ 96.3%, you have two one-sided choices of intervals. Either [0.9400, 0.9733] or [0.94, 0.99].

I assume your $n$ will be much greater than 3, so you will resample with replacement. Say you do this 1000 times. Then find the mean in each case. From the set of 1000 means, take the middle 950 values. The lowest and highest values of this subset form the 95% confidence interval.

The question here: How do we create a confidence interval for the parameter of a permutation test? gives more detail, including some R code.

$\endgroup$
  • $\begingroup$ As written in another comment, n will not be "much greater than 3", but maybe n=10 is possible if needed. While this approach guarantees that my confidence interval will not go beyond 1.0, it seems to considerably underestimate the confidence interval given by other methods. In fact, it will never be larger than the [min,max] interval. $\endgroup$ – koalo Feb 24 '17 at 7:02
  • $\begingroup$ How often do you think the mean will be outside of [min,max]? $\endgroup$ – soakley Feb 24 '17 at 18:54
  • $\begingroup$ Probably rarely, but does that also mean that if the [min,max] interval is small enough to prove support my claims I can forget about the confidence interval and just provide [min,max]? In my experience, for small sample sizes, the confidence interval is rather large compared to [min,max]. $\endgroup$ – koalo Feb 24 '17 at 19:23
2
$\begingroup$

Binomial confidence intervals have been the subject of statistician debates for a long time. Your problem considers a less than 100% ratio, but it becomes even more problematic if we use 100%. One insightful way to ask the question is:

Given the sun has risen without fail every day for the past 2,000 years, what is the probability that it will rise tomorrow?

With such a high success rate, we think the chances are pretty high, but we can't be 100% sure (the universe might explode first, or something). So, even if you had a 100% proportion, we can't let the confidence interval collapse at $p=1$.

There are a number of methods to calculate these tails. I'd recommend checking out Wikipedia for the math, or if you just want the answer, search for a binomial interval calculator like this one (which happens to also have some more explanation of the math behind it).

$\endgroup$
  • $\begingroup$ That is very near to what I am looking for, but the formulas only seem to calculate the confidence interval for the outcome of a single run of my experiment and not a confidence interval for the mean of several experiments. $\endgroup$ – koalo Feb 23 '17 at 9:21
  • $\begingroup$ It doesn't matter if you have one run or several runs, as long as the denominator (100 packets in you example) remains the same in all runs. Running 3 experiments of 100 each is mathematically the same as running one experiment with 300 packets, and you can use the binomial formulas, but with n=300 and not n=100. If the denominators are not equal, you need to find the weighted mean (weighted by the n's) and the new n will be the sum of the n's. $\endgroup$ – Zahava Kor Feb 23 '17 at 13:13
  • $\begingroup$ @ZahavaKor Since it is too long for a comment, I added an edit to my question. I don't say it is wrong, but it does not match my current understanding. $\endgroup$ – koalo Feb 23 '17 at 17:06
2
$\begingroup$

A Bayesian approach:

Find the unique beta distribution $B$ that is induced by the experiments (and a prior, say, the Jeffreys prior), and then choose the smallest interval for which $B$'s density integrates to your desired "confidence". It's possible for there to be multiple solutions, and depending on your prior, the mean ratio might not be in your interval.

$\endgroup$
  • $\begingroup$ +1, even though this would not be a confidence interval, but a credible interval. Can you say a bit more about how to find a beta distribution? One can start with a flat prior Beta(1,1), but how to update it given a set of observations such as e.g. {0.985,0.986,0.935,0.890,0.999}? One usually uses Beta as a conjugate to Binomial, and there the updating for each observed $p=n/m$ is easy, but how to update given $p$ only? $\endgroup$ – amoeba says Reinstate Monica Feb 27 '17 at 22:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.