0
$\begingroup$

I want to measure the distribution of a population's performance on a test. Each person takes a version of the test with a random selection of N questions from a large pool of possible questions. There is uncertainty in this score as an estimate of their "true score" on the set of all possible questions, which I model using a beta distribution for each person (with the parameters being the counts of incorrect answers and correct answers, plus 1 for a uniform prior).

Now I want to combine each person's score to find the distribution of scores for the population, while propagating uncertainty correctly. How should I combine my measures of individuals' scores, along with their beta-distributed uncertainty, into a population measure?

Further thought: As I see it, the beta distributions around each individual's score represent epistemic uncertainty, while the distribution of scores represents intrinsic variability in the population, and I'm not sure about the best/correct way of incorporating the epistemic uncertainty from the individual level into the population estimate. Could this be done with a hierarchical model, where the individual's beta distribution parameters are themselves drawn from a population distribution? Hope this makes sense..!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.