2
$\begingroup$

Considering a scenario where a tool can detect errors in a program. The result in each try is whether an error was detected or not. Suppose the tool detected 30 errors in 50 trials.

I calculate this confidence interval like:

1.96 * squareroot(0.6*(1-0.6)/50)

Sample Size: 50

Observed Proportion: 60%

Confidence Level: 95%

Confidence Interval: ±12.52

Range for the true population proportion: 53.48% to 78.52%

And now I was trying to understand the results but Im not understanding one thing.

For example I can say that if we run the expriment n times (i think 50 times) we expcet to obtain a probability of error detection between 53.48% and 78.52 95% of the times. But how can we do so its possible to say "if we run the expriment n times we expect to obtain between x and y number of errors?" We need to use a different formula to reach this conclusion?

$\endgroup$

2 Answers 2

4
$\begingroup$

To interpret the confidence interval, you must assume that the n=50 observations you made were randomly chosen from a much larger (infinitely large) population. The question is what is the fraction of errors in that population.

You can say that you are 95% confident that the range from 53.48% to 78.52% includes that population proportion.

More precisely, you can say that if you repeated your n=50 experiment many times (no reason to think about 50 repetitions as a special value), you'd expect to get different confidence intervals each time (or most of the time) and expect that 95% of the computed confidence intervals will include the true population value and 5% not to.

Note that your equation to compute a confidence interval is an approximation that always creates symmetrical intervals. Better methods (that create asymmetrical intervals) are available.

$\endgroup$
1
$\begingroup$

You are using the wrong school of thought to answer that question. Although Frequentist "prediction intervals" exist, they lack the property of coherency, which means you should not invest money or gamble on them.

How I am answering this depends entirely on how you worded it. Had you worded it just a little differently, I would have answered this differently. I am hoping I have a reasonable understanding of what you are asking. Just for clarity, what you are asking is for the ability to say that if you would run the experiement n more times, then you will see between $j_1$ and $j_2$ errors $z\%$ of the time.

That is sort of easy to solve.

Let me back up and explain why I am changing schools of thought. The school you are using is called the Frequentist school. The two other major schools are the Likelihoodist and the Bayesian. The Bayesian is the oldest. It separates inference from decision into a two step process. It depends upon what information you already have about the problem at hand. The Likelihoodist school, which you cannot use because it has no intervals, tries to discover new knowledge. The Frequentist school is behavioral. If you reject the null then you behave as if it is false and treat the minimum variance unbiased estimator as if the true value. If the null is not rejected, then you behave as if the null is true, regardless if it is or not. It is behavioral in nature. It tells you how to behave.

The Bayesian school is about finding things. In this case you want to find both the location of the parameter and find the probability the values are between some set of boundaries.

A Bayesian solution has four parts, the prior distribution, the likelihood, the evidence and the posterior distribution. There is also a fifth part that you are interested in, the predictive distribution.

The difficulty is that there are infinite number of Bayesian solutions and they depend upon what you already know about the problem, before seeing the data. If this were a well studied problem with understood properties, then the solution would be totally different than if you were totally ignorant of its properties.

We will start the problem in a state of total ignorance. We will assume that you believe that the true value of $p$ is equally probable anywhere between 0 and 1. Your prior distribution is 1 when $0<p<1$ and 0 everywhere else. You could use less than or equal too as well, but I am ruling out the trivial case.

The likelihood is just the binomial distribution, which is $\binom{n}{k}p^k(1-p)^{n-k}$ This is divided by the integral $\int_0^1\binom{n}{k}p^k(1-p)^{n-k}\mathrm{d}p$. Setting $c$ to the constant of integration, which in this case $c=2,403,589,824,441,960$, your posterior distribution is $c(1-p)^{20}p^{30}$, for all $p$ in the set $0<p<1$.

This is the likely distribution for the true value of $p$, not if the observed percentage errors. I suggest plotting this, just so you can get a visualization. I also suggest solving this in logs and then converting back. The boundaries of your 95% highest density region for the parameter is (.461,.724). Notice that these are different. It also has a different meaning. The meaning of the 95% confidence interval is that if you would repeat the experiment n times, then at least 95% of the intervals that are created will contain the true value of the parameter, for a randomly selected sample. This is a Bayesian credible interval. It says there is a 95% chance the true value of the parameter is inside the set. It says nothing about what happens if you repeat it.

In its broad general form $$Pr(p|n,k,{\alpha},{\beta})=\binom{n}{k}\frac{1}{B({\alpha},{\beta})}p^{k+{\alpha}-1}(1-p)^{n+{\beta}-k-1},$$ where $\alpha$ and $\beta$ parameterize your prior distribution. In the above example $\alpha=\beta=1$. The function $B({\alpha},{\beta})$ is shortened from $\frac{({\alpha}-1)!(\beta{-1})!}{(\alpha{+}\beta{-1})!}$.

Now as to the prediction. You want to know the probability of seeing $k'$ successes in the next $n'$ tests. To do this you need to integrate out the remaining uncertainty at each possible number for the observed presence of at least one software error.

In general the solution is $Pr(k'|n')=\int_0^1\binom{n'}{k'}p^{k'}(1-p)^{n'-k'}\Pr(p|n,k,{\alpha},\beta{)}\mathrm{d}p$. This has also been solved so that your prediction for each discrete value $k'$ in the set from 1 to $n$ is $$\binom{n'}{k'}\frac{B(k+k'+\alpha{,}n'+n-k-k'+{\beta})}{B(k+\alpha{,}n-k+\beta{)}}$$.

Your >95% predicted interval for 50 repetitions is between 19 of 50 and 39 of 50 trials to have errors. It is greater than 95% because these are discrete counts. As a percentage, you expect between 38% and 78% of the next fifty observations with a 96.4% probability. This is the 96.4% credible set of predictions.

The difficulty of this method is there is no table you can use to look this up. There is no spreadsheet that has been preconstructed. You have to do this on your own. If you do not know calculus, but do know factorials then you can use the last formula to solve this problem.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.