2
$\begingroup$

Let's pretend that in a Financial / IT / Construction Audit of a task which the outcome is either Pass(1) or Fail(0).

Suppose that the Auditee should perform this task on a daily basis (365 days in a year), what would be the formula / test that the Auditor can use to find the minimum sample size (in days) that supports such statement: "The Auditor believes, with 95% confidence, that the Auditee has Passed 90% of daily task performed in 1 year. "

Appreciated if the answer could use the parameters above as example.

Without digging too deep into the formula or tables, my attempt using this tool is as follows: https://stattrek.com/survey-sampling/sample-size-calculator.aspx

Input

Sampling method: Simple random sampling

Parameter of interest: Proportion

Purpose of research: Test hypothesis

Main goal: Find min sample size, given error

Margin of error: 0.05

Significance level: 0.05

Null hypothesis: Proportion >= 0.95

Total population size: 365

Output

Sample size = 46

Region of acceptance = 0.900 to 1.000

======

Does a sample size of 46 support the statement above?

$\endgroup$
6
  • $\begingroup$ I apologize that your question was mistakenly closed. $\endgroup$
    – whuber
    Commented May 19, 2020 at 11:24
  • $\begingroup$ @base64, What is the total population size and what is the effect size? $\endgroup$ Commented May 19, 2020 at 13:56
  • $\begingroup$ @DhamnekarWinod Population size is 365 and there is only single group. $\endgroup$
    – base64
    Commented May 19, 2020 at 14:18
  • $\begingroup$ @base64, You have not stated your problem in clear way. $\endgroup$ Commented May 19, 2020 at 14:38
  • $\begingroup$ I reopened the question because, even in its original form, it looked perfectly clear to me. $\endgroup$
    – whuber
    Commented May 19, 2020 at 15:12

1 Answer 1

2
$\begingroup$

If audit was conducted 365 times, what is the probability that it would pass 365*0.9 times? The probability can be calculated from Binomial distribution $B(k|365,p)$. The problem is that $p$ is unknown. So, we need a trick.

First, we estimate it by conducting $n$ trials and get $\hat p=\frac{m+1}{n+2}$, where $m$ was the number of times audit passed in $n$ trials. Here, I'm using the Laplace's rule of succession. The reason is that if you have $n$ successes then usual MLE estimate gives $\hat p=1$ and it messes up the bootstrapping step.

Second, we need a sampling distribution of $\hat p$. We can either an approximation or an "exact" one. There are many approximations, e.g. using Wald method we get the sampling distribution: $$f(\tilde p)=\hat p+ z\sqrt{\hat p(1-\hat p)/n},$$ where z is the standard normal.

The "exact" method isn't really exact but it's a bootstrapping from $\hat p$: we plug it into the binomial distribution, generate $n$ trials and get $\tilde p=\tilde m/n$, where $\tilde m$ is number of passes. This is what "exact" method is based on.

Third, we convolve $f(\hat p)$, either approximation or "exact", with the binomial distribution: $$k\sim \int_{0}^1 B(k|365,\hat p)f(\hat p)d \hat p$$ Where $k$ is the number of times passed in 365 audits.

Once you got the distribution, perhaps numerically, you can estimate the probability $Pr[k>365*0.9]$ and get the probability that you were looking for.

Now, you should see that your question doesn't have a unique answer. In other words, you pick any $n$ and produce a probability that auditee would have passed 90% of time in daily audits during the year. The only things that will change is the power of the test, it will increase with $n$, but you didn't specify the required power.

Example: suppose you conducted n=46 audits and 42 of them passed. This means $\hat p=0.896$, and the sampling distribution is $f(\hat p)=\mathcal N(0.896,0.045)$ in Wald method or in "exact" method you get the corresponding Binomial distribution. Monte Carlo simulation with 1000 trials produces the probability of passing 329 or more time in 365 trials is $\pi\approx 47\pm 1.6$% in Wald and $\pi\approx 35\pm 1.5$% in "exact" method. The Monte Carlo uncertainty is $\sqrt{\pi*(1-\pi)/1000}\approx 1.5$%. Here's a cumulative histogram for Wald and "exact" methods: enter image description here enter image description here

VERY IMPORTANT CASE: ongoing audit

The previous discussion was based on the notion that you did $n$ trials this year, and planning for the next year audit. What if we're planning the remaining audit? In other words $n$ is a part of 365 days of audit. So we cunducted $n$ audits and now are thinking how many more we need before we're confident that auditee would pass 90% of total 365 audits?

In this case you have to make an important adjustment to the step 3 in the previous discussion. You only generate 365-$n$ trials in Monte Carlo if you're using this method. Here's what I got with "exact" method for this case and the same n=46 and m=43. The estimate of probability is the same $\hat p=0.896$. So we run Monte Carlo with this setting but generate only passes for 365-46 trails and add the passed to earlier 42 passed. Then we build the cumulative distribution of total passes. In this case the probability of 90% passes in total 365 trials changes a bit: $\pi\approx 38\pm1.5$%. Here's a histogram: enter image description here

This modification doesn't make a lot of difference for low initial trial numbers $n$. However, as $n$ grows it becomes very important. Consider the case where you made already $n=364$ trials with 328 successes. In this case the probability of getting 90% successes is equal $\hat p$, because we have essentially a Bernoulli experiment at hand at this point. Unlike the case where you're planning for the next year, where you didn't run any audit at all yet.

This is how you do it in Excel. First you calculate the estimated $\hat p$ in cell F6 using the Laplace formula. Then you bootstrap the sampling distribution of $\tilde p$ in column B: enter image description here

Then for each realization of $\tilde p$ in column B you simulate remianing trials and show obtained passes k in column C: enter image description here

Finally, you count in how many simulations you passed the required mark of 90% in 365 audits: enter image description here

$\endgroup$
14
  • $\begingroup$ Hi Aksakal, thanks for your in-depth answer. Would your answer vary if k is always coincidentally >= np before such tests are run? $\endgroup$
    – base64
    Commented May 19, 2020 at 16:26
  • $\begingroup$ @base64, do you mean to say that in the past they always passed the threshold? I dont understand your question $\endgroup$
    – Aksakal
    Commented May 19, 2020 at 16:35
  • $\begingroup$ Hi Aksakal, yes, given that they have passed the threshold within the sample. For example, if I sampled 46 days in a year and found that more than 41.4 (e.g. 42) is a "pass", can I safely conclude that 328.5 of 365 days is also a "pass", without looking at the 319 days (the rest of the population). $\endgroup$
    – base64
    Commented May 19, 2020 at 16:43
  • $\begingroup$ @base64, I updated my answer with your example in comments, but the answer is No, it's much lower than 90% probability they'll pass 329 times or more. If they passed 44 times, then we'd be talking! Such a drastic sensitivity tells you that the power of a test is low at n=46 $\endgroup$
    – Aksakal
    Commented May 19, 2020 at 16:54
  • $\begingroup$ @Aksakal, Original post is asking what is minimum sample size? Stattrek calculator provided 102 days as a sample size. What is your answer for the required sample size?:-) $\endgroup$ Commented May 20, 2020 at 3:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.