0
$\begingroup$

I have a question about sample size calculations under the Bayesian setting, and would like either some ideas/suggestions, or reference papers that might help.

In general, my problem is as follows. I have a population of computer parts of which some proportion of the parts will be defective. I would like, say, for no more than 1% of the parts to be defective. Here, I consider a part to be defective if the corresponding measured value $X$ > 10. Now, if I have, say, 1,000,000 of these computer parts then I could test 99% of them to see if the defect rate is higher than 1% (which I don't want to do). However, I want to only test a subset of the parts to be able to make the "no more than 1% defective parts claim" with some level of certainty.

Now, I would like to approach this problem from the Bayesian point-of-view, and assume that the distribution of measurements $X$ follows some distribution (say, Normal($\mu$, $\sigma^2$)) and associated prior $p(\mu,\sigma^2)$. Under the Bayesian paradigm how does one proceed with sample size estimation for this scenario? I.e., how do I choose a sample size such that I have some level of certainty that the $Pr(X>10) < 0.01$, for example.

Also, I don't want to treat the variable as binary since I do have continuous measurements $X$ that I can leverage.

Improve this question
$\endgroup$
14
  • 2
    $\begingroup$ Pretty much the only way to be certain that $Pr(X>10)<0.01$ is to choose the prior so that this holds, and then to not draw any sample, because the sample may always bring up observations with $X>10$ that can "spoil" this probability. I guess being "certain that $Pr(X>10)$ is whatever" is not what you really want to achieve, because in fact you want to find out what the data have to say about $Pr(X>10)$, don't you? $\endgroup$
    – Christian Hennig
    Dec 13, 2021 at 23:49
  • $\begingroup$ @ChristianHennig Yes sorry. Perhaps certain is a bit loose here. Maybe it should have been more probabilistic. Like how can I choose the sample size such that the probability that Pr(X>10) < 0.01 is no larger than $1-\alpha$. I basically want high probability that the statement is true. $\endgroup$
    – John Smith
    Dec 13, 2021 at 23:52
  • 1
    $\begingroup$ I'm not an expert in Bayesian sample size calculations, however I think you want something like what you state in your comment above assuming that indeed in your number of 1,000,000 parts the relative frequency of defective parts is at most $\beta$, and note that $\beta$ may need to be smaller than 0.01, because if $\beta=0.01$, then it will probably not be possible to get the probability for $Pr(X>10)\ge 0.01$ arbitrarily small. $\endgroup$
    – Christian Hennig
    Dec 14, 2021 at 0:00
  • $\begingroup$ The baseline is (not sure whether you understand this already) that obviously you should only be able to get a high probability that your statement is true, if indeed there is only a less than 0.01 proportion of defective parts, otherwise not. $\endgroup$
    – Christian Hennig
    Dec 14, 2021 at 0:03
  • $\begingroup$ @ChristianHennig yes this makes sense to me but thank you for pointing out the baseline. $\endgroup$
    – John Smith
    Dec 14, 2021 at 0:05

1 Answer 1

Reset to default
0
$\begingroup$

From a Bayesian perspective, you just want to compute $P(X > 10)$ given a series of measurements of $X$. There may not exist a sample size for which $P(X>10)<0.01$ if $P(X>10)$ is in fact greater than 0.01! For example, you take 10,000 measurements of $X$ and they're all greater than 10. You'd have to have a pretty skewed prior to conclude $P(X>10)<0.01$. In general, $X>10$ is a binary event, and computing $P(X>10)$ is typically performed via a Beta distribution. You can find an open source implementation of this exact functionality here, with an accompanying introduction and tutorial. While those are geared toward evaluating performance metrics, the same process applies to your problem.

UPDATE: The pdf you cited is asking "Given a prior on $P(X>10)$, what sample size should I expect to need to verify that $P(X > 10) < 0.01$ with some higher degree of certainty (technically they're asking how many required to narrow margin of error of $P(X > 10)$ by some amount, but I think that's the best way to translate that source onto your problem). This effectively gives rise to a compound experiment:

  1. Sample a possible P(X > 10) from the prior. 1
  2. Simulate hypothetical future samples until your posterior estimate of $P(X > 10)$ (prior combined with new synthetic samples) is less than 0.01. Record the number of synthetic samples you needed to reach that conclusion.
  3. Repeat steps 1 and 2 and build a histogram of how many synthetic samples you needed.

From there, you could pick the most common number of samples needed, or the upper $\alpha$ quantile of your histogram of samples (i.e. $\alpha$ chance that this sample size will be sufficient to establish $P(X > 10) < 0.01$ with some threshold of certainty). The pdf you linked seems to effectively describe ways to streamline that procedure combined with different ways to extract a sample size from that histogram. But I think the core concept is accurately outlined with the above experiment.

That said, this may or may not be the optimal approach to your particular problem. If you have to set out funding for a large, expensive, time consuming study ahead of time, sure you'd want to try to figure out what sample size you'd need to verify a claim expected to be true to some greater degree of certainty than you have. However, if you can get feedback as you go, you could effectively just run one real experiment instead of a large $N$ synthetic ones from step 1 above. That is, just keep measuring parts until your posterior estimate (as described in my first answer) of $P(X > 10) < 0.01$ falls below your chosen threshold of certainty.

1 In the paper, this would be a distribution implied from a previous study. In your example, it's your best idea of what the distribution of $P(X > 10)$ is. You'd typically use a beta distribution for this as described above. For example, if you recall having witnessed 10 working parts, and one nonfunctional one, you could use Beta(10, 1). See https://en.wikipedia.org/wiki/Additive_smoothing#Pseudocount for an overview of this heuristic. This distribution represents your uncertainty of $P(X > 10)$, but since it's expressed as a probability distribution, you can sample it. This simulates possible real world values of $P(X > 10)$ based on your current understanding.

Improve this answer
$\endgroup$
15
  • $\begingroup$ Maybe I'm not being clear in my question. This is just a sample size question before collecting any data. How many samples do I need to collect to feel sufficiently confident that the true defect rate is < 0.01 (maybe it is, maybe it isn't). Is it, 10?, 100, all 1,000,000? Frequentist do this all the time with power calculations, but I don't see how to do this from the Bayesian point-of-view. $\endgroup$
    – John Smith
    Dec 14, 2021 at 15:11
  • $\begingroup$ I think you're confusing p values with an estimate of a statistic. Frequentists compute the probability of results "at least as extreme" (for some definition of extreme left to the statistician) as the ones observed occurring from a random sample. That's all a p-value is. See en.wikipedia.org/wiki/Misuse_of_p-values for more on that. Bayesians try to quantify their certainty of possible values of parameters like P(X>10) using probability distributions. They would therefore attempt to compute P(P(X > 10) < 0.01) in a justifiable way. $\endgroup$
    – Alex Eftimiades
    Dec 14, 2021 at 15:21
  • $\begingroup$ The point of my question isn't to do what frequentist do, but to understand how the Bayesian does sample size determination. I think you are missing my point. There has to be a way to choose the sample size such that some criteria is satisfied. That is what I need help with and or wanted suggestions for. Check out a book on Bayesian reliability and you'll see plenty of Bayesian sample size calculations. $\endgroup$
    – John Smith
    Dec 14, 2021 at 15:51
  • $\begingroup$ The first thing I get when I google "Bayesian reliability" is itl.nist.gov/div898/handbook/apr/section4/apr46.htm, which does not mention sample size calculations. Is there a particular reference you had in mind? $\endgroup$
    – Alex Eftimiades
    Dec 14, 2021 at 17:08
  • $\begingroup$ link.springer.com/book/10.1007/978-0-387-77950-8 $\endgroup$
    – John Smith
    Dec 14, 2021 at 17:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.