3
$\begingroup$

I wanted to do some exercises to improve my basic stats skills, but the following simple problem from Gelman's "applied regression" exam made me think quite a bit.

A multiple-choice test item has four options. Assume that a student taking this question either knows the answer or does a pure guess. A random sample of 100 students take the item. 60% get it correct. Give an estimate and 95% confidence interval for the percentage in the population who know the answer

(with solution and discussion here)

I read the solution and I have understood it, but I can't really understand why this first thought I had is wrong.

Let $X$ be the proposition "student knows the answer" and $Y$ "student got the correct answer". Assuming as a prior $P(X)=1/2$, and knowing $P(Y|X)=1, P(Y|\neg X)=1/4$ we can compute $$P(Y) = P(Y|X)P(X) + P(Y|\neg X)P(\neg X) = 5/8$$ Then by Bayes theorem $$P(X|Y) = P(Y|X)P(X)/P(Y) = 4/5$$ Since the observed percentage of students who got the answer correct is 0.6, my point estimate for the percentage of population who know the answer would be 0.6*(4/5)=0.48, which is different from the correct result 0.47

At first I thought that this was because I took a "bayesian route" to the solution, so I checked a proper (at least I think so) bayesian model where $X$ is the number of students who know the answer (discrete uniform from 0 to 100), and $Y$ is the number of students who got the correct answer. However, running the following code I got the correct result

from scipy.special import binom
import numpy as np


def likelihood(x): 
    if x < 0 or x > 60:
        return 0
    return binom(100-x,60-x)*(0.25**(60-x))*(0.75**(40))

def compute_posterior():
    results = []
    for x in range(101):
        results += [likelihood(x)*(1/101)]
    return np.array(results)/sum(results)

post = compute_posterior()
print("MAP: ",np.argmax(post)) # 47!

Can you please help me understand why the first simple argument is wrong?

$\endgroup$
3
  • 1
    $\begingroup$ Your probability statements don't make sense to me. You need P(X), the probability that a student knows the answer. You're calculating P(X|Y), the probability student knows the answer given they got the question correct. $\endgroup$
    – Eli
    Feb 3, 2020 at 15:20
  • $\begingroup$ For that matter, multiplying your posterior by the fraction of people in the class doesn't answer the posed question - you're supposed to estimate the probability in the population, not in the classroom. $\endgroup$
    – jkm
    Feb 3, 2020 at 15:34
  • $\begingroup$ Yes, but since 0.6 in my best estimate that a member of the population gets the answer correct, shouldn't 0.48 be the best estimate for the probability that one knows the answer (following my reasoning at the individual level)? @jkm $\endgroup$
    – Alessandro
    Feb 3, 2020 at 20:22

1 Answer 1

5
$\begingroup$

I believe it's because your priors are very different between the two scenarios. To see this, we first need to put the priors on the same interpretation (The first scenario talks about an individual, the the second scenario talks about a population).

In the first scenario, your prior on a student knowing a correct answer is $P(X)=0.5$. If we extend this to a population of 100 students, the prior on the number of students who know the right answer is $\text{Bin}(100, 0.5)$.

In the second scenario, your prior on the number of students who know the right answer is $\text{Uni}(0, 100)$.

Because of the effect of the prior, your posterior MAP will be higher in the first scenario than the second.

See the following simulation, and note the two priors:

from scipy.special import binom
from scipy.stats import binom as rbinom
import numpy as np


def likelihood(x): 
    return binom(100-x,60-x)*(0.25**(60-x))*(0.75**(40))

def prior1(x):
    return rbinom(100, 0.5).pmf(x)

def prior2(x):
    return 1/101

def compute_posterior(prior):
    results = []
    for x in range(101):
        results += [likelihood(x)*prior(x)]
    return np.array(results)/sum(results)

post = compute_posterior(prior1)
print("MAP: ",np.argmax(post)) # 48!

post = compute_posterior(prior2)
print("MAP: ",np.argmax(post)) # 47!

$\endgroup$
5
  • $\begingroup$ Thank you! Now I see I wrongly assumed that maximal ignorance at individual level and at sample level should lead to the same answer. However I'm not sure to understand the reason on your emphasis on the number of students who get it right vs my *number of students who know the correct answer". Shouldn't I put a prior on the variable of which I want the posterior? Second question: following my reasoning a the individual level is it wrong to assume that the same 0.48 should be my posterior estimate also for the out-of-sample population in the same way the original (Gelman's) solution does? $\endgroup$
    – Alessandro
    Feb 3, 2020 at 20:31
  • $\begingroup$ whoops, my italics are wrong, and it should be "know the correct answer"! $\endgroup$ Feb 4, 2020 at 1:29
  • $\begingroup$ I didn't see any discussion about out-of-sample in the url you posted. Maybe you can expand on that? $\endgroup$ Feb 4, 2020 at 1:36
  • $\begingroup$ Sorry, what I mean is: are those inference I carried out (and you corrected) appropriate to estimate the percentage in the population who know the answer? Or should they be interpreted only as the percentage of respondents (i.e. in the collected random sample) who know the answer? $\endgroup$
    – Alessandro
    Feb 4, 2020 at 8:34
  • 1
    $\begingroup$ This gets at the generalizability (also known as transportability) of your experiment. You can't answer that using the info you provided. For example, maybe the sample was from doctors, who obviously know more about medical questions than the average population. So to directly answer your question, your probably limited to "percentage of respondents" $\endgroup$ Feb 4, 2020 at 13:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.