I was following up on this problem when it seemed to me that the correct answer was ambiguous. I will not re-state the problem since it can be found in the original link. I will use the same notation conventions as the accepted answer, namely that $M_1$ will designate the model for having mastery and$M_2$ for not having mastery. $N$ and $c$ indicate the number of questions and correct answers, respectively.

To be more explicit, where I am having difficulty, I'll pickup from where the accepted answer finished. We have the ratio of model posteriors:

$\frac{p(M_1|c)}{p(M_2|c)} \geq 19 $

By plugging in using Bayes theorem, we have:

$\frac{p(c | M_1)p(M_1)}{p(c | M_2)p(M_2)} \geq 19 $

The likelihoods $p(M_i|c)$ are binomials and $p(M_i)$ are the given priors. By substituting with numerical values given in the problem, we get:

$ \frac{{N \choose c} (1/2)^c(1 - 1/2)^{N-c}\times (1/2)} {{N \choose c} (1/4)^c (1-1/4)^{N-c} \times (1/2)} \geq 19 $

This can be simplified to:

$ \frac{(1/2)^N}{(1/4)^c \times (3/4)^{N-c}} \geq 19 $.

However, at this point, we cannot directly solve for $N$, since $c$ is also a variable. What is the correct approach for determining the smallest possible value of $N$?


Best-case Scenario

The smallest possible value of $N$ where we could be $95\%$ confident that the student has "mastered" a concept (meaning that their probability of answering a question correctly is $1/2$) would correspond to picking $c = N$, because $P(M_1 | c)$ is an increasing function of $c$, and $c$ is constrained by $c \le N$. In this case, $$ \frac{(1/2)^N}{(1/4)^c \times (3/4)^{N-c}} = \frac{(1/2)^N}{(1/4)^N} = 2^N $$ and the smallest value of $N$ that makes $2^N \ge 19$ is $N = 5$.

Anti-best-case Scenario

If we wanted to be $95\%$ confident that a student has not mastered the material, then the best-case scenario would be $c = 0$ where the student misses every question. Then we have $$ \frac{(1/2)^N}{(1/4)^c \times (3/4)^{N-c}} = \frac{(1/2)^N}{(3/4)^N} = (2/3)^N $$ and the smallest value of $N$ for which $(2/3)^N \le 1/19$ is $N = 8$, in which case we can be $95\%$ confident that the student has not mastered the material.

Worst-case Scenario

Assuming that the student has a fixed probability $\delta$ of getting the questions right, in the limit as $N \to \infty$ we have $c \approx \delta N$. Taking $\log_2$ of both sides of the desired inequality we have \begin{align} \frac{(1/2)^N}{(1/4)^c \times (3/4)^{N-c}} & \ge 19 \\ N \log_2(1/2) - c\log_2(1/4) - (N - c)\log_2(3/4)& \ge \log_2 (19) \\ N \log_2(1/2) + c\log_2(4) + (N - c)\log_2(4/3)& \ge \log_2 (19) \\ -N + 2c + (N - c)(2 - \log_2(3)) & \ge \log_2 (19) \\ -N + 2\delta N + (1 - \delta)(2 - \log_2(3))N & \ge \log_2 (19) \\ N[-1 + 2\delta + (1 - \delta)(2 - \log_2(3))] & \ge \log_2 (19) \\ N[1 - \log_2(3) + \log_2(3) \delta] & \ge \log_2 (19). \end{align} Because the expression $1 - \log_2(3) + \log_2(3) \delta$ is 0 when $$\delta = \frac{\log_2 (3) - 1}{\log_2 (3)} \approx 0.36907,$$ we can never be $95\%$ confident either way even if $N \to \infty$ for a student that gets exactly this fraction of questions correct. This means that there is no universally valid value $N$ that will always ensure $95\%$ confidence either that the student is a master or is not a master. Notice that this value is very close to the midpoint between $1/2$ and $1/4$, which is $0.375$. Intuitively, it is difficult to determine whether a student that gets $\approx 37\%$ of questions correct is a master with probability $P(\text{correct}) = 1/2$ or not a master with probability $P(\text{correct}) = 1/4$.

| cite | improve this answer | |
  • $\begingroup$ Great response @ericperkerson, thank you! The only thing I don't entirely understand is how you knew to set 1−log2(3)+log2(3)𝛿 to 0 in the Worst-case Scenario. Is it because you are specifically looking for a case where the inequality would never make sense, regardless of N? $\endgroup$ – slacker Sep 4 at 15:04
  • $\begingroup$ Well if $1 - \log_2 (3) + \log_2 (3) \delta = 0$, the last line of the algebra above then says that $N \times 0 \ge \log_2 (19)$, which can never be true regardless of how large $N$ is. I didn't know this value before doing that algebra though (I would have guessed beforehand that the value was the midpoint $1/2(1/2 + 1/4) = 0.375$, which is very close but not quite correct). And yes, I was specifically looking for a case where the inequality would never be satisfied. $\endgroup$ – Eric Perkerson Sep 4 at 17:05
  • 1
    $\begingroup$ Thanks again. Unfortunately, I don't have high enough reputation to mark yours as an accepted answer but I absolute would if I could. $\endgroup$ – slacker Sep 6 at 19:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.