1
$\begingroup$

This question relates to my previous query regarding differences in IRT parameter estimates (2PLM model) between the ltm package in R and the Mplus latent variable program which is linked here: 2PLM IRT modeling of rare event behavioral data: Why changing discrimination and difficulty values?

Now having obtained similar parameter estimates across platforms and having a sense of reliability to my estimates, I want to have a better grasp of what one considers 'plausible' discrimination and difficulty (threshold) estimate values in 2PLM IRT modeling. As is the case with any data, this will obviously hinge on sample size, response patterns etc. However, because I am dealing with unique maternal behaviors that are very rare-- even in a child protective setting-- many of my binary indicators have extreme cuts, in some cases with only a 1% proportion of occurrence (i.e., score = 1) on a sample size of N = 343. Despite this, all of my models successfully converge and the best loglikelihood is replicated--and in cases where the best LL is not replicated, contending solutions present essentially identical parameter estimates.

However, for some dimensions assessed, I get extremely large discrimination and difficulty estimates (e.g., well beyond the -3 to +3 prototypical scale of theta, or the latent trait under consideration) along with extremely large standard errors (in parens). Take, for example, data for the following binary indicators of two different latent traits:

Indicator 1: discrimination = 25.71(547.95), difficulty = .60 (1.44)

Indicator 2: discrimination = -.11 (0.45), difficulty = -30.58 (130.42)

Obviously, these are very large values for the discrimination parameter (indicator 1) and the difficulty parameter (indicator 2) and the SEs are even more extreme. Nevertheless, the models converge and given the inherent nature of the data, I am inclined to treat these values as accurate estimates of the sample under consideration. As to their degree of precision, this will have to await replication efforts but given the laborious nature of coding such data, I imagine it is likely going to be some time before another dataset of this nature and size is assembled. With all that in mind, I am interested in recommendations or 'best practice' guidelines in such situations assuming limitations of the data and precision of the estimates are openly acknowledged.

This paper on sparse data bias with maximum likelihood estimators was useful in many ways, but did not quite suit my needs in the current situation http://www.bmj.com/content/352/bmj.i1981

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

With standard errors being so wide, I'm not sure I would consider this model as 'converged' (the fact that different software agree is really just a numerical test, not a test of whether the model is actually unique or even useful). The problem with such large SEs is that parameter estimates are essentially interchangeable, in reality you have no idea where the true population values are (in your example, you can't even be sure the discrimination is positive!).

The question should come down to how much do these items actually contribute to your test information to improve measurement precision. Discrimination parameters that are so large are essentially degenerate because they represent a perfect inflection at a given $\theta$ location but nothing more, while extremely difficult/easy items only provide information about individuals who have extreme $\theta$ values (which is generally very rare). They also tend to influence the quality of other parameters in the model, and therefore should be removed or given a strong prior parameter distribution to lessen their influence and keep them closer to a priori more reasonable values.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Is it not the case that in the 2PLM model, the item information is a function of the discrimination parameter, thus items with the highest discrimination values will contain the most information at the level corresponding to the items difficulty (or in my case what I am referring to as severity) parameter? I ask as I am using the item information curves to retain the 2 most informative items at the severe end (more generally the positive end) of the trait. $\endgroup$
    – Jhaltiga68
    Commented Jun 7, 2016 at 20:43
  • 1
    $\begingroup$ @Jhaltiga68 sure, but in both these cases neither is helpful because you have no idea where the population parameters are. There is so little confidence in the population values that the standard errors just report complete uncertainty; hence, they tell you next to nothing about how useful the items are. $\endgroup$
    – philchalmers
    Commented Jun 7, 2016 at 21:22
  • $\begingroup$ So would a reasonable strategy, then, be to remove such items with nonsensical standard errors from the latent trait analyses and rerun the model without that item? The issue to me--from a more conceptual standpoint--is, assuming that the behaviors are indeed extremely rare (to observe), could we not make a case that, despite such absurd standard error values, our point estimate is 'the best we can do' at approximation given the rarity of the data? $\endgroup$
    – Jhaltiga68
    Commented Jun 8, 2016 at 2:53
  • 1
    $\begingroup$ I would say yes removing those items is the best strategy. While I see your issue, in terms of measurement more generally those items aren't the most useful for 99.99% of the people who would be measured. Keeping those extreme items in with a 'best we can do' approach really doesn't add anything either, because you still will be measuring the extreme traits with a large degree of uncertainty. Unless you had multiple items around the same difficulty for accurately measuring these persons, one/two items will make little to no difference. $\endgroup$
    – philchalmers
    Commented Jun 8, 2016 at 5:08
  • $\begingroup$ Been working some more with the problematic models and see that I am not replicating the best LL although both ltm and Mplus "converge" on a solution. Clearly the data sparsity is part of the problem but I am thinking through how best to deal with it. I tried deleting some of the offending items, but the models still remain problematical. Nothing seems too off in the bivariate residuals though with or without the items with nonsensical estimates...any other ideas how to diagnose what might be going on? $\endgroup$
    – Jhaltiga68
    Commented Jun 8, 2016 at 5:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.