# Determining sample size from a Bayesian problem

I was following up on this problem when it seemed to me that the correct answer was ambiguous. I will not re-state the problem since it can be found in the original link. I will use the same notation conventions as the accepted answer, namely that $$M_1$$ will designate the model for having mastery and$$M_2$$ for not having mastery. $$N$$ and $$c$$ indicate the number of questions and correct answers, respectively.

To be more explicit, where I am having difficulty, I'll pickup from where the accepted answer finished. We have the ratio of model posteriors:

$$\frac{p(M_1|c)}{p(M_2|c)} \geq 19$$

By plugging in using Bayes theorem, we have:

$$\frac{p(c | M_1)p(M_1)}{p(c | M_2)p(M_2)} \geq 19$$

The likelihoods $$p(M_i|c)$$ are binomials and $$p(M_i)$$ are the given priors. By substituting with numerical values given in the problem, we get:

$$\frac{{N \choose c} (1/2)^c(1 - 1/2)^{N-c}\times (1/2)} {{N \choose c} (1/4)^c (1-1/4)^{N-c} \times (1/2)} \geq 19$$

This can be simplified to:

$$\frac{(1/2)^N}{(1/4)^c \times (3/4)^{N-c}} \geq 19$$.

However, at this point, we cannot directly solve for $$N$$, since $$c$$ is also a variable. What is the correct approach for determining the smallest possible value of $$N$$?

## Best-case Scenario

The smallest possible value of $$N$$ where we could be $$95\%$$ confident that the student has "mastered" a concept (meaning that their probability of answering a question correctly is $$1/2$$) would correspond to picking $$c = N$$, because $$P(M_1 | c)$$ is an increasing function of $$c$$, and $$c$$ is constrained by $$c \le N$$. In this case, $$\frac{(1/2)^N}{(1/4)^c \times (3/4)^{N-c}} = \frac{(1/2)^N}{(1/4)^N} = 2^N$$ and the smallest value of $$N$$ that makes $$2^N \ge 19$$ is $$N = 5$$.

## Anti-best-case Scenario

If we wanted to be $$95\%$$ confident that a student has not mastered the material, then the best-case scenario would be $$c = 0$$ where the student misses every question. Then we have $$\frac{(1/2)^N}{(1/4)^c \times (3/4)^{N-c}} = \frac{(1/2)^N}{(3/4)^N} = (2/3)^N$$ and the smallest value of $$N$$ for which $$(2/3)^N \le 1/19$$ is $$N = 8$$, in which case we can be $$95\%$$ confident that the student has not mastered the material.

## Worst-case Scenario

Assuming that the student has a fixed probability $$\delta$$ of getting the questions right, in the limit as $$N \to \infty$$ we have $$c \approx \delta N$$. Taking $$\log_2$$ of both sides of the desired inequality we have \begin{align} \frac{(1/2)^N}{(1/4)^c \times (3/4)^{N-c}} & \ge 19 \\ N \log_2(1/2) - c\log_2(1/4) - (N - c)\log_2(3/4)& \ge \log_2 (19) \\ N \log_2(1/2) + c\log_2(4) + (N - c)\log_2(4/3)& \ge \log_2 (19) \\ -N + 2c + (N - c)(2 - \log_2(3)) & \ge \log_2 (19) \\ -N + 2\delta N + (1 - \delta)(2 - \log_2(3))N & \ge \log_2 (19) \\ N[-1 + 2\delta + (1 - \delta)(2 - \log_2(3))] & \ge \log_2 (19) \\ N[1 - \log_2(3) + \log_2(3) \delta] & \ge \log_2 (19). \end{align} Because the expression $$1 - \log_2(3) + \log_2(3) \delta$$ is 0 when $$\delta = \frac{\log_2 (3) - 1}{\log_2 (3)} \approx 0.36907,$$ we can never be $$95\%$$ confident either way even if $$N \to \infty$$ for a student that gets exactly this fraction of questions correct. This means that there is no universally valid value $$N$$ that will always ensure $$95\%$$ confidence either that the student is a master or is not a master. Notice that this value is very close to the midpoint between $$1/2$$ and $$1/4$$, which is $$0.375$$. Intuitively, it is difficult to determine whether a student that gets $$\approx 37\%$$ of questions correct is a master with probability $$P(\text{correct}) = 1/2$$ or not a master with probability $$P(\text{correct}) = 1/4$$.

• Great response @ericperkerson, thank you! The only thing I don't entirely understand is how you knew to set 1−log2(3)+log2(3)𝛿 to 0 in the Worst-case Scenario. Is it because you are specifically looking for a case where the inequality would never make sense, regardless of N? – slacker Sep 4 at 15:04
• Well if $1 - \log_2 (3) + \log_2 (3) \delta = 0$, the last line of the algebra above then says that $N \times 0 \ge \log_2 (19)$, which can never be true regardless of how large $N$ is. I didn't know this value before doing that algebra though (I would have guessed beforehand that the value was the midpoint $1/2(1/2 + 1/4) = 0.375$, which is very close but not quite correct). And yes, I was specifically looking for a case where the inequality would never be satisfied. – Eric Perkerson Sep 4 at 17:05
• Thanks again. Unfortunately, I don't have high enough reputation to mark yours as an accepted answer but I absolute would if I could. – slacker Sep 6 at 19:41