Simple exercise in point estimation: what did I do wrong?

Question

I wanted to do some exercises to improve my basic stats skills, but the following simple problem from Gelman's "applied regression" exam made me think quite a bit.

A multiple-choice test item has four options. Assume that a student taking this question either knows the answer or does a pure guess. A random sample of 100 students take the item. 60% get it correct. Give an estimate and 95% confidence interval for the percentage in the population who know the answer

(with solution and discussion here)

I read the solution and I have understood it, but I can't really understand why this first thought I had is wrong.

Let $X$ be the proposition "student knows the answer" and $Y$ "student got the correct answer". Assuming as a prior $P(X)=1/2$, and knowing $P(Y|X)=1, P(Y|\neg X)=1/4$ we can compute $$P(Y) = P(Y|X)P(X) + P(Y|\neg X)P(\neg X) = 5/8$$ Then by Bayes theorem $$P(X|Y) = P(Y|X)P(X)/P(Y) = 4/5$$ Since the observed percentage of students who got the answer correct is 0.6, my point estimate for the percentage of population who know the answer would be 0.6*(4/5)=0.48, which is different from the correct result 0.47

At first I thought that this was because I took a "bayesian route" to the solution, so I checked a proper (at least I think so) bayesian model where $X$ is the number of students who know the answer (discrete uniform from 0 to 100), and $Y$ is the number of students who got the correct answer. However, running the following code I got the correct result

from scipy.special import binom
import numpy as np


def likelihood(x): 
    if x < 0 or x > 60:
        return 0
    return binom(100-x,60-x)*(0.25**(60-x))*(0.75**(40))

def compute_posterior():
    results = []
    for x in range(101):
        results += [likelihood(x)*(1/101)]
    return np.array(results)/sum(results)

post = compute_posterior()
print("MAP: ",np.argmax(post)) # 47!

Can you please help me understand why the first simple argument is wrong?

Answer 1

I believe it's because your priors are very different between the two scenarios. To see this, we first need to put the priors on the same interpretation (The first scenario talks about an individual, the the second scenario talks about a population).

In the first scenario, your prior on a student knowing a correct answer is $P(X)=0.5$. If we extend this to a population of 100 students, the prior on the number of students who know the right answer is $\text{Bin}(100, 0.5)$.

In the second scenario, your prior on the number of students who know the right answer is $\text{Uni}(0, 100)$.

Because of the effect of the prior, your posterior MAP will be higher in the first scenario than the second.

See the following simulation, and note the two priors:

from scipy.special import binom
from scipy.stats import binom as rbinom
import numpy as np


def likelihood(x): 
    return binom(100-x,60-x)*(0.25**(60-x))*(0.75**(40))

def prior1(x):
    return rbinom(100, 0.5).pmf(x)

def prior2(x):
    return 1/101

def compute_posterior(prior):
    results = []
    for x in range(101):
        results += [likelihood(x)*prior(x)]
    return np.array(results)/sum(results)

post = compute_posterior(prior1)
print("MAP: ",np.argmax(post)) # 48!

post = compute_posterior(prior2)
print("MAP: ",np.argmax(post)) # 47!