I have developed a recent interest in Item Response Theory and its applications. I am studying clinical psychology and am most interested in polytomous models aimed at modelling psychopathology (e.g., a depression measure that uses likert scales). Specifically, I am interested in how certain measures can discriminate between individuals with and without a certain mental disorder. Typically, I would examine this using exploratory and/or confirmatory factor analyses and follow up with examinations of specificity and sensitivity.

I understand that in item response theory and its applications, it is important that a heterogeneous sample is used. This allows all potential answers on a question to be endorsed by a least some participants (e.g., all five options on a Likert scale; Embretson & Reise, 2000)0. I also appreciate that this would allow relatively wider inferences about the population of interest.


Is it appropriate to use only a clinical sample when applying item response theory to determine if a measure is useful in discriminating between individuals with a certain mental disorder, and those without?


My intuition tells me that it is not appropriate, as only a relatively small fraction of the population would be tested (e.g., 12% of the population with social anxiety disorder). As such, it is my inclination to believe such an analysis would only allow us to know how those with the clinical disorder would respond, and not those without. This would limit the ability of the measure to truly assess the range of psychopathology, which would be useful for discriminating purposes. Correct?

Here is a link to a specific article that does this: http://www.ncbi.nlm.nih.gov/pubmed/21744971

The authors of this article manually inspected ICC curves in a social anxiety disorder sample, deleted items that did not have an ICC curve that was near 45 degrees (I'm assuming this is appropriate... They argued it demonstrated that each point on the Likert scale added more information. On a side note, I would also appreciate some thoughts on this approach!), and subsequently used the new measure do discriminate between those with and without social anxiety disorder. As I mentioned, I believe this is problematic, but it is published in a highly respectable journal and I find little information to answer my questions. Note that this measure would typically be given to a general population and would be used as a diagnostic or screening tool, and that social anxiety has been argued as on a continuum (i.e., dimensional from minimal to severe) rather than taxonic or categorical (i.e., clinically severe vs normative).

One potential obstacle that I see is that when I run ICC curves with only individuals with social anxiety disorder, the curves are near 45 degrees, and when I do so with a heterogeneous sample (i.e., both social anxiety disorder and "healthy" controls), the curves are quite distorted. Note that these are only the combined ICC curves for all items and not a polytonomous plot that includes curves for each of the response options. I hope this paragraph makes sense...

Any answers or guidance would be appreciated! Also, relevant references would be of great use to me!

  • $\begingroup$ Hello @Behacad, I had a quick look at the abstract of the article in question, and they didn't just sample from individuals with social anxiety disorder. It appears that they had 3-cohorts: 1) socially phobic patients enrolled in 5-treatment trials (n=456), 2) an independent clinical sample (n=?), and 3) an independent student sample (n=?). So it appears that they might have had a heterogeneous enough sample to get item statistics that would be invariant. I am going to run out of room here, but will try to put my thoughts in order and place them in the longer answer format. $\endgroup$ – doug.numbers May 1 '13 at 22:22
  • $\begingroup$ I am missing on the 45 degree lines -- why is that important? $\endgroup$ – StasK Jun 13 '13 at 23:48

There are two sets of questions that are relevant here.

  1. Does the model work the same way for different people and different groups in the population? E.g. if males and females have different slopes with respect to their "ability" (in IRT slang; for your case, this would the degree of social anxiety), then that would be bad for an instrument, as it would give different picture for males and females. With ordinal items, you may need to worry that people with different degree of anxiety may have different thresholds, which is again bad for the instrument. Psychometricians investigate that under the title of differential item functioning (DIF), and you may want to check some intro reading on that topic. Heterogeneity of the sample is important here to the extent that different groups complete the instrument, and you have enough data to identify the problems with DIF, if any.
  2. Identification: assuming that your instrument works exactly the same way for everybody, you need to ensure that all categories have been endorsed on all items. Only using the high "ability"/anxiety people to calibrate the instrument will likely make it impossible to figure out what the thresholds for the ordinal scale are on its low end, and that's why you need enough controls that will rarely endorse the higher values on the scale. Once again, heterogeneity here is important, but it hinges on absence of DIFs.

On a side note, you should not trust every published result these days. Your judgement may be as good as the reviewers' judgement... especially if that journal has contacted you to write a referee report for them... especially if you did it more than once.

| cite | improve this answer | |

@Behacad, Building on the comment above, I think the answer is that the article in question was appropriate. Please check out the following link, as Dr. Panayides used the Rasch model and a polytomous set of data. I think you will find his methods of interest and that they will provide the information you need. All the best!

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.