1
$\begingroup$

I'm doing some statistics on multiple choice test scores (with item response theory, (http://en.wikipedia.org/wiki/Item_response_theory). I want to know the probability that a student with a certain grade (say between 70-80% on the rest of the test) got a specific question right, and the associated uncertainty to this probability.

Say I have 160 students that got between 70-80% on the test, and that 113 of these got the question I'm looking at right. The chance of another student in the same bin getting the question right is 113/160 -- but what is the uncertainty of this measurement?

$\endgroup$

1 Answer 1

0
$\begingroup$

It's not usually the test % that is focused on in IRT, it's usually based on the latent trait scores. So you may want to think about how a particular observed score (e.g. 113) relates to the latent trait. Once you have an estimate of that, you'll know the general probabilities for each item by plugging them into the estimated IRT parameters.

Uncertainty is also related to $\hat{\theta}$ in the form $SE(\hat{\theta})$ should help to answer the last part of your query. Of course this also relates to item-parameter uncertainty as well since less precise parameters will contribute to the overall uncertainty in the expected probabilities, but that's a different story.

$\endgroup$
2
  • $\begingroup$ My goal is to find some way to measure the latent trait score that isn't dependent on the test, but I'm having trouble figuring out a good way, so for now I'm just using the test scores (minus the specific question I'm looking at). But yeah, I agree. As for your second part -- what do you mean by SE(theta)? Sorry, I'm quite new at this. By the way, what I'm using now is a binomial distribution centered around 113 (num. corr. ans), with the probability being 113/150. I know this isn't ideal, though, because it assumes the probability I find is accurate, and collapses when P(theta) = 0 or 1. $\endgroup$
    – Patrick
    Commented Oct 16, 2014 at 22:08
  • $\begingroup$ Using the test scores is actual one kind of approximation for $\hat{\theta}$, so you are generally getting at what you want there. If you collect all the sum scores and standardized them (i.e., convert to z-scores) this is one crude approach. Much like other latent variable models though it is subject to measurement error, which is where the $SE(\theta)$ term comes in (IRT has a specific meaning for this). Determining model implied standard errors will help you form intervals for the probabilities since you can evaluate the upper/lower CIs of $P(\hat{\theta})$ given the item parameters. $\endgroup$ Commented Oct 16, 2014 at 22:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.