I want to measure the distribution of a population's performance on a test. Each person takes a version of the test with a random selection of N questions from a large pool of possible questions. There is uncertainty in this score as an estimate of their "true score" on the set of all possible questions, which I model using a beta distribution for each person (with the parameters being the counts of incorrect answers and correct answers, plus 1 for a uniform prior).

Now I want to combine each person's score to find the distribution of scores for the population, while propagating uncertainty correctly. How should I combine my measures of individuals' scores, along with their beta-distributed uncertainty, into a population measure?

Further thought: As I see it, the beta distributions around each individual's score represent epistemic uncertainty, while the distribution of scores represents intrinsic variability in the population, and I'm not sure about the best/correct way of incorporating the epistemic uncertainty from the individual level into the population estimate. Could this be done with a hierarchical model, where the individual's beta distribution parameters are themselves drawn from a population distribution? Hope this makes sense..!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.