4
$\begingroup$

I really like how traditional item response theory (IRT) packages tell you the standard error of measurement conditional on one's ability level, and from that, you can calculate the test information function and convert that to a conditional reliability-like metric. However, most IRT packages are limited to dichotomous or categorical variables.

I'm aware that IRT with continuous responses is just factor analysis, but I don't think traditional factor analysis packages tell you how the reliability / measurement error of your factor score changes across ability levels, which is what I want to find out. Is this possible, and are there any R packages that do this?

Improve this question
$\endgroup$
5
  • $\begingroup$ In IRT, the concept you're referring to is information, specifically the Fisher Information. For example, the 2-parameter logistic model's item information function is, $I(\theta) = a_i^2p_iq_i$, where $p$ and $q$ are the probability of answering the question correctly and wrongly, and $a$ is the discrimination parameter. The whole test's information function is the sum of all item information functions. And $SE(\theta) = \frac{1}{\sqrt I(\theta)}$. We need the information function for the general SEM case. Sadly, providing this is a bit beyond my ability (see what I just did there?). $\endgroup$
    – Weiwen Ng
    Commented Mar 19, 2019 at 19:21
  • $\begingroup$ Note that asking for R packages is off topic here. You may not get an answer to that. $\endgroup$
    – gung - Reinstate Monica
    Commented Mar 19, 2019 at 20:42
  • $\begingroup$ I don't think this exists. A dichotomous score only provides information about one part of the scale - a very hard question gives you no information to distinguish between average and below average respondents. A continuous scale should distinguish between people all along the scale - unless it's highly skewed, which violates an assumption. So information is equal at all parts of the scale. $\endgroup$
    – Jeremy Miles
    Commented Mar 20, 2019 at 17:08
  • $\begingroup$ @JeremyMiles We know in IRT that the standard error at various levels of $\theta$ is a property of the Fisher information. Won't a linear SEM model also have Fisher information, or is that quantity not defined for some reason? $\endgroup$
    – Weiwen Ng
    Commented Mar 20, 2019 at 17:44
  • 2
    $\begingroup$ Also, imagine that we do have a bunch of tests of various difficulty whose responses are distributed as Gaussian. Maybe it's something like completion time for a puzzle, and most people score around some mean. Imagine these puzzles vary in difficulty. Now, imagine that you give a very capable person an easy puzzle. They'll complete it very fast. But, as with standard IRT models, I would posit that the puzzle I described provides very little info about their latent trait. I would find it a bit odd if a linear SEM provided equal info at all values of the scale. $\endgroup$
    – Weiwen Ng
    Commented Mar 20, 2019 at 17:46

1 Answer 1

Reset to default
2
$\begingroup$

I don't see anybody jumping to answer this, so I'm going to turn my comment into an answer despite not having a complete solution to this. However, the poster is asking about the standard error of measurement in a non-item response theory (IRT) model. Because he or she is referring to a continuous response, I have to assume the indicator variables are Gaussian, and that this is an application of linear structural equation modeling.

IRT models take binary or ordinal items (i.e. questions). Each item provides some information about the latent trait. For example, in a 2-parameter logistic model, the $i$-th question has this information function:

$I(\theta) = a_i^2p_i(1-p_i)$

where $p_i$ is the model-estimated probability of responding correctly to the $i$-th question at a certain level of $\theta$, and $a_i$ is the question's discrimination parameter. The entire test's information function is the sum of all the individual item information function. Then, the standard error at any given $\theta$ is $SE(\theta) = 1 - \frac{1}{\sqrt{I(\theta)}}$.

Now, information is the Fisher information. All we need is the Fisher information for each item in the linear SEM case. Unfortunately, that is beyond my ability* to provide. I think it's the expectation of the first derivative of the log likelihood with respect to $\theta$. Can anyone else provide any insight?

A possible alternative is this: I know for sure that Stata will estimate both the value and the standard error of each observation's latent trait. I have to think that R packages like lavaan have to do the same (but I haven't tried this in lavaan). You might be able to plot latent trait vs standard error. In IRT, this may not work as well, because there are going to be a finite number of predicted latent trait values and the graph will look a bit chunky. If you genuinely have continuous indicators, you may have a finer distribution of the latent trait estimates.

As a side note: Stata gets around this by creating a simulated dataset with 300 values of $\theta$ from -4 to 4, and then calculating item and test information at each value of $\theta$. I've basically replicated this process to produce a graph for a non-standard IRT model. I believe that the R package mirt may do something similar.

*See what I just did there?

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.