0
$\begingroup$

In our online system, we run 1000 tests (where each test presents one piece of functionality) and found 30 bugs/errors, i.e. 30 tests failed.
My background in statistic is pretty much non-existent, so I would like to know whether the following make sense:
I believe I could estimate - assuming the normal distribution - the proportion of errors occurring while using the system.

Using the following: z0,975=1,96 (I want to be "95% confident") p=30/1000 n=1000

p-sqrt( p(1-p)/n * z < π < p+sqrt( p(1-p)/n * z

As a result, could I say that in 95% of time there is between X and X1% functions that fail?

$\endgroup$

1 Answer 1

0
$\begingroup$

Your formula is correct, at least according to what is taught in most classical textbooks. One catch, however, is contained in your phrase "assuming the normal distribution" --- this approximation may be quite poor even for relatively large sample sizes:

Interval Estimation for a Binomial Proportion (Brown, Cai, DasGupta, 2001)

Fortunately (see the cited paper) there are simple adjustments that make the confidence intervals have a coverage probability closer to what you specify (in your case 95 per cent). I do not fully comprehend your last sentence with the "X", but it lets me suspect that the following explanation may be useful to you: the confidence interval changes from sample to sample, while the unknown true proportion is fixed. Thus the correct reading is something like "the [random] confidence interval covers the true proportion about 95 per cent of the time in when sampling repeatedly".

In any case, the test outcomes ought to be at least approximately independent for the normal approximation to work.

$\endgroup$
1
  • $\begingroup$ Thanks! The X simply refers to the result (I did not calculate it). $\endgroup$
    – John V
    Dec 13, 2017 at 10:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.