Idea or reference for sample size calculation

Question

I have a question about sample size calculations under the Bayesian setting, and would like either some ideas/suggestions, or reference papers that might help.

In general, my problem is as follows. I have a population of computer parts of which some proportion of the parts will be defective. I would like, say, for no more than 1% of the parts to be defective. Here, I consider a part to be defective if the corresponding measured value $X$ > 10. Now, if I have, say, 1,000,000 of these computer parts then I could test 99% of them to see if the defect rate is higher than 1% (which I don't want to do). However, I want to only test a subset of the parts to be able to make the "no more than 1% defective parts claim" with some level of certainty.

Now, I would like to approach this problem from the Bayesian point-of-view, and assume that the distribution of measurements $X$ follows some distribution (say, Normal($\mu$, $\sigma^2$)) and associated prior $p(\mu,\sigma^2)$. Under the Bayesian paradigm how does one proceed with sample size estimation for this scenario? I.e., how do I choose a sample size such that I have some level of certainty that the $Pr(X>10) < 0.01$, for example.

Also, I don't want to treat the variable as binary since I do have continuous measurements $X$ that I can leverage.

Answer 1

From a Bayesian perspective, you just want to compute $P(X > 10)$ given a series of measurements of $X$. There may not exist a sample size for which $P(X>10)<0.01$ if $P(X>10)$ is in fact greater than 0.01! For example, you take 10,000 measurements of $X$ and they're all greater than 10. You'd have to have a pretty skewed prior to conclude $P(X>10)<0.01$. In general, $X>10$ is a binary event, and computing $P(X>10)$ is typically performed via a Beta distribution. You can find an open source implementation of this exact functionality here, with an accompanying introduction and tutorial. While those are geared toward evaluating performance metrics, the same process applies to your problem.

UPDATE: The pdf you cited is asking "Given a prior on $P(X>10)$, what sample size should I expect to need to verify that $P(X > 10) < 0.01$ with some higher degree of certainty (technically they're asking how many required to narrow margin of error of $P(X > 10)$ by some amount, but I think that's the best way to translate that source onto your problem). This effectively gives rise to a compound experiment:

1. Sample a possible P(X > 10) from the prior. 1
2. Simulate hypothetical future samples until your posterior estimate of $P(X > 10)$ (prior combined with new synthetic samples) is less than 0.01. Record the number of synthetic samples you needed to reach that conclusion.
3. Repeat steps 1 and 2 and build a histogram of how many synthetic samples you needed.

From there, you could pick the most common number of samples needed, or the upper $\alpha$ quantile of your histogram of samples (i.e. $\alpha$ chance that this sample size will be sufficient to establish $P(X > 10) < 0.01$ with some threshold of certainty). The pdf you linked seems to effectively describe ways to streamline that procedure combined with different ways to extract a sample size from that histogram. But I think the core concept is accurately outlined with the above experiment.

---

That said, this may or may not be the optimal approach to your particular problem. If you have to set out funding for a large, expensive, time consuming study ahead of time, sure you'd want to try to figure out what sample size you'd need to verify a claim expected to be true to some greater degree of certainty than you have. However, if you can get feedback as you go, you could effectively just run one real experiment instead of a large $N$ synthetic ones from step 1 above. That is, just keep measuring parts until your posterior estimate (as described in my first answer) of $P(X > 10) < 0.01$ falls below your chosen threshold of certainty.

1 In the paper, this would be a distribution implied from a previous study. In your example, it's your best idea of what the distribution of $P(X > 10)$ is. You'd typically use a beta distribution for this as described above. For example, if you recall having witnessed 10 working parts, and one nonfunctional one, you could use Beta(10, 1). See https://en.wikipedia.org/wiki/Additive_smoothing#Pseudocount for an overview of this heuristic. This distribution represents your uncertainty of $P(X > 10)$, but since it's expressed as a probability distribution, you can sample it. This simulates possible real world values of $P(X > 10)$ based on your current understanding.