0
$\begingroup$

Under Item Response Theory, test information $I(\hat\theta)$ is a function of the examinee's estimated ability $\hat\theta$, discovered a the end of the test, and the items that were answered during said test. The standard error of estimation $SE$, in turn, is a function of the test information. I would like to calculate $SE$ in a CAT, but first I need to calculate the test information.

Question: In a computerized adaptive test (CAT), how do you calculate test information if, at each step of the test, a new value of $\hat\theta$ is estimated? Do we use the final estimation, like this: $$I(\hat\theta) = \sum_{0<t<N} I_t(\hat\theta_N)$$ or do we use the partial estimations of $\theta$ during the test to calculate item information at that particular point, like $$I(\hat\theta) = \sum_{0<t<N} I_t(\hat\theta_t)$$

Improve this question
$\endgroup$
5
  • $\begingroup$ Not sure I understand your question. The SEE definition here is the expected standard error, while the standard error of $\hat{\theta}$ is based on the observed responses. You can use SSE as the SE estimate of $\theta$, but for estimators other than ML it won't be correct (e.g., using a Bayesian estimator the SEE term will not be appropriate). $\endgroup$
    – philchalmers
    Feb 13, 2016 at 19:54
  • $\begingroup$ I think I used the wrong $\theta$ in my $SEE$. If we calculate the standard error using $I(\hat{\theta})$ instead of $I(\theta)$, can it then be considered IRT ordinary standard error? Also, thanks for the edit. Once this pre-question is out of the way, I'll elaborate more in my question. $\endgroup$
    – Douglas De Rizzo Meneghetti
    Feb 13, 2016 at 22:34
  • $\begingroup$ You have to use $\hat{\theta}$, you never actually have access to the population parameter in order to calculate the true information. The information function can be used as a suitable standard error when the observed information is identical to the expected information (which for 2PL and GRM's is true) and only when the estimator is found via maximum-likelihood. $\endgroup$
    – philchalmers
    Feb 14, 2016 at 1:33
  • $\begingroup$ I guess my question is better now. If I find a way to calculate a CAT information, I can then calculate its standard error. $\endgroup$
    – Douglas De Rizzo Meneghetti
    Feb 14, 2016 at 3:17
  • $\begingroup$ Information (test or item) are just functions, so it doesn't matter if $\hat{\theta}$ changes during your estimation. Just change the respective inputs to obtain the desired values. Perhaps where you are getting hung up is that information functions assume that $\theta$ is fixed and known, where the trick in CATs is to both select informative items and improve the potential accuracy of $\hat{\theta}$ at the same time. $\endgroup$
    – philchalmers
    Feb 14, 2016 at 4:08

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.