I have developed a recent interest in Item Response Theory and its applications. I am studying clinical psychology and am most interested in polytomous models aimed at modelling psychopathology (e.g., a depression measure that uses likert scales). Specifically, I am interested in how certain measures can discriminate between individuals with and without a certain mental disorder. Typically, I would examine this using exploratory and/or confirmatory factor analyses and follow up with examinations of specificity and sensitivity.
I understand that in item response theory and its applications, it is important that a heterogeneous sample is used. This allows all potential answers on a question to be endorsed by a least some participants (e.g., all five options on a Likert scale; Embretson & Reise, 2000)0. I also appreciate that this would allow relatively wider inferences about the population of interest.
Is it appropriate to use only a clinical sample when applying item response theory to determine if a measure is useful in discriminating between individuals with a certain mental disorder, and those without?
My intuition tells me that it is not appropriate, as only a relatively small fraction of the population would be tested (e.g., 12% of the population with social anxiety disorder). As such, it is my inclination to believe such an analysis would only allow us to know how those with the clinical disorder would respond, and not those without. This would limit the ability of the measure to truly assess the range of psychopathology, which would be useful for discriminating purposes. Correct?
Here is a link to a specific article that does this: http://www.ncbi.nlm.nih.gov/pubmed/21744971
The authors of this article manually inspected ICC curves in a social anxiety disorder sample, deleted items that did not have an ICC curve that was near 45 degrees (I'm assuming this is appropriate... They argued it demonstrated that each point on the Likert scale added more information. On a side note, I would also appreciate some thoughts on this approach!), and subsequently used the new measure do discriminate between those with and without social anxiety disorder. As I mentioned, I believe this is problematic, but it is published in a highly respectable journal and I find little information to answer my questions. Note that this measure would typically be given to a general population and would be used as a diagnostic or screening tool, and that social anxiety has been argued as on a continuum (i.e., dimensional from minimal to severe) rather than taxonic or categorical (i.e., clinically severe vs normative).
One potential obstacle that I see is that when I run ICC curves with only individuals with social anxiety disorder, the curves are near 45 degrees, and when I do so with a heterogeneous sample (i.e., both social anxiety disorder and "healthy" controls), the curves are quite distorted. Note that these are only the combined ICC curves for all items and not a polytonomous plot that includes curves for each of the response options. I hope this paragraph makes sense...
Any answers or guidance would be appreciated! Also, relevant references would be of great use to me!