I wanted to do some exercises to improve my basic stats skills, but the following simple problem from Gelman's "applied regression" exam made me think quite a bit.
A multiple-choice test item has four options. Assume that a student taking this question either knows the answer or does a pure guess. A random sample of 100 students take the item. 60% get it correct. Give an estimate and 95% confidence interval for the percentage in the population who know the answer
(with solution and discussion here)
I read the solution and I have understood it, but I can't really understand why this first thought I had is wrong.
Let $X$ be the proposition "student knows the answer" and $Y$ "student got the correct answer". Assuming as a prior $P(X)=1/2$, and knowing $P(Y|X)=1, P(Y|\neg X)=1/4$ we can compute $$P(Y) = P(Y|X)P(X) + P(Y|\neg X)P(\neg X) = 5/8$$ Then by Bayes theorem $$P(X|Y) = P(Y|X)P(X)/P(Y) = 4/5$$ Since the observed percentage of students who got the answer correct is 0.6, my point estimate for the percentage of population who know the answer would be 0.6*(4/5)=0.48, which is different from the correct result 0.47
At first I thought that this was because I took a "bayesian route" to the solution, so I checked a proper (at least I think so) bayesian model where $X$ is the number of students who know the answer (discrete uniform from 0 to 100), and $Y$ is the number of students who got the correct answer. However, running the following code I got the correct result
from scipy.special import binom import numpy as np def likelihood(x): if x < 0 or x > 60: return 0 return binom(100-x,60-x)*(0.25**(60-x))*(0.75**(40)) def compute_posterior(): results =  for x in range(101): results += [likelihood(x)*(1/101)] return np.array(results)/sum(results) post = compute_posterior() print("MAP: ",np.argmax(post)) # 47!
Can you please help me understand why the first simple argument is wrong?