12
$\begingroup$

I am in the process of empirically developing a questionnaire and I will be using arbitrary numbers in this example to illustrate. For context, I am developing a psychological questionnaire aimed at assessing thought patterns commonly identified in individuals who have anxiety disorders. An item could look like "I need to check the oven repeatedly because I can't be sure its off".

I have 20 questions (5-point Likert) which may be comprised of one or two factors (note that in reality I have closer to 200 questions, comprised of 10 scales, and each scale may be comprised of two factors). I am willing to erase about half the items, leaving 10 questions on one of two factors.

I am familiar with exploratory factor analysis (EFA), internal consistency (Cronbach's alpha), and item characteristic curves in item response theory (IRT). I can see how I would use any single of these methods to determine which items are the "worse" within any single scale. I appreciate that each method also answers different questions, although they may lead to similar results and I am not sure what "question" is most important.

Before we start, lets make sure I know what I am doing with each of these methods individually.

  • Using EFA, I would identify the number of factors, and remove the items that load the least (lets say <.30) on their respective factor or that cross-load substantially across factors.

  • Using internal consistency, I would remove items that have the worse "alpha if item deleted". I could do so assuming one factor in my scale, or do it after an initial EFA to identify the number of factors and subsequently run my alpha for each factor.

  • Using IRT, I would remove items that do not assess the factor of interest along their (5 Likert) response options. I would be eyeballing item characteristic curves. I would basically be looking for a line on a 45 degree angle going from option 1 on the Likert scale all the way up to 5 along the latent score. I could do so assuming one factor, or do it after an initial
    EFA to identify number of factors, and subsequently run the curves for each factor.

I am unsure which of these methods to use in order to best identify which items are the "worst". I use worst in a broad sense such that the item would be of detriment to the measure, either in terms of reliability or validity, both of which are equally important to me. Presumably I can use them in conjunction, but I am not sure how.

If I was to go ahead with what I know now and give it my best shot I would do the following:

  1. Do an EFA to identify number of factors. Also delete items with bad loadings on their respective factors, since I don't want items that load badly regardless of how they would do in other analyses.
  2. Do IRT and remove bad items judged by that analysis as well, if any remain from the EFA.
  3. Simply report Cronbach's Alpha and don't use that metric as a means to delete items.

Any general guidelines would be greatly appreciated!

Here is also a list of specific questions that you can perhaps answer:

  1. What is the practical difference between removing items based on factor loadings and removing items based on Chronbach's alpha (assuming you use the same factor layout for both analyses)?

  2. Which should I do first? Assuming I do EFA and IRT with one factor, and both identify different items that should be removed, which analysis should have priority?

I am not hard set on doing all of these analyses, although I will report Chronbach's alpha regardless. I feel like doing just IRT would leave something missing, and likewise for just EFA.

$\endgroup$
  • $\begingroup$ If you choose to achieve construct validity via FA you should of course start with FA (after screening out items with "bad", e.g. too skewed distributions). Your engagement with FA will be complex and iterative. After throwing out most "weak" items, rerun FA, check KMO index, degree of restoration of correlations, factor interpretability, check if more items to delete, then rerun again $\endgroup$ – ttnphns Jul 17 '12 at 20:15
  • $\begingroup$ Using Classical Test Theory in Combination with Item Response Theory is a good read. $\endgroup$ – chl Jul 17 '12 at 20:29
  • 1
    $\begingroup$ You remove the items with the highest "alpha if item removed" not lowest... $\endgroup$ – user29835 Sep 3 '13 at 6:03
  • $\begingroup$ It's strange! as to this basic question, we have no a recognized answer within 3 years. $\endgroup$ – WhiteGirl Jul 26 '17 at 15:18
3
$\begingroup$

I don't have any citations, but here's what I'd suggest:

Zeroth: If at all possible, split the data into a training and test set.

First do EFA. Look at various solutions to see which ones make sense, based on your knowledge of the questions. You'd have to do this before Cronbach's alpha, or you won't know which items go into which factor. (Running alpha on ALL the items is probably not a good idea).

Next, run alpha and delete items that have much poorer correlations than the others in each factor. I wouldn't set an arbitrary cutoff, I'd look for ones that were much lower than the others. See if deleting those makes sense.

Finally, choose items with a variety of "difficulty" levels from IRT.

Then, if possible, redo this on the test set, but without doing any exploring. That is, see how well the result found on the training set works on the test set.

$\endgroup$
  • $\begingroup$ Thanks for the answer. This is along the direction I was thinking, although I am not sure if I will have the cases to split the data. Also, since the items are on 5-point Likert scales, I expect most of them, or at least the "good ones", will exhibit similar difficulty. $\endgroup$ – Behacad Jul 17 '12 at 20:24
  • 1
    $\begingroup$ Surely, you know good references :-) I'd tease you on the following points (because this thread will likely serve as a reference for future questions). (a) Usually, item deletion based on Cronbach's alpha is done without considering a cross-validation scheme. Obviously, it is a biased approach as the same individuals are used to estimate both measures. (b) Another alternative is to base item/scale correlation by considering rest score (that is, sum score without including the item under consideration): do you think it matters in this case? (...) $\endgroup$ – chl Jul 17 '12 at 20:29
  • 1
    $\begingroup$ (...) (c) Finally, IRT models are often used to discard items (in the spirit of scale purification) based on item fit statistics and the like. What's your opinion on that approach? $\endgroup$ – chl Jul 17 '12 at 20:29
  • $\begingroup$ FYI I can probably find references for each of these methods individually, but I would appreciate any potential references to using any of these methods in conjunction. Any references would be great, really! You know (and probably are!) reviewers... $\endgroup$ – Behacad Jul 17 '12 at 21:28
  • $\begingroup$ @chl I could dig up references, but I don't know them off the top of my head. On a) and b), it probably matters more than most people think it does; someone should do a simulation. on c) Been a while since I did IRT stuff (my degree is in psychometrics, but that was long ago). $\endgroup$ – Peter Flom - Reinstate Monica Jul 17 '12 at 21:45
2
$\begingroup$

All three of your suggested criteria actually could be performed in IRT, more specifically multidimensional IRT. If your sample size is fairly large would probably be a consistent way to go about it for each subscale. In this way you could get the benefits of IRT for modelling item independently (using nominal models for some items, generalized partial credit or graded for others, or if possible even set up rating scales to help interpret polytomous items in a more parsimonious way).

MIRT is conceptually equivalent to item-level factor analysis and therefore has a linear EFA equivalent relationship for dichotomous and polytomous items. I'm not sure if I would buy into the <.3 criteria for dropping items though, since it really depends on the context and factor structure. Small loadings/slopes don't provide as much information about the intercept locations, but may still be useful since they can offer a wider and less peaked information function across levels of $\theta$. Some applications in CAT make use of these types of items early on as well since they give a wider band of information early on in the test.

Dropping items based on the Cronbach criteria is roughly the same as dropping items that give a better marginal/empirical reliability in IRT, so if the software you are using supports these statistics then you could follow the same strategy without leaving the IRT paradigm. I'd be more inclined to check the information functions however to see if removing an item severely affects the measurement at various $\theta$ levels (related to where the intercepts are). Relative information plots are useful here as well.

You could attempt to remove items that don't conform to the unidimensional requirements of most IRT software, but I wouldn't necessarily recommend this if it affects the theoretical representation of the constructs at hand. In empirical applications it's usually better to try and make our models fit our theory, not the other way around. Also, this is where the bifactor/two-tier models tend to be appropriate since you would like to include all possible items while accounting for multidimensionality in a systematic and theoretically desirable way.

$\endgroup$
  • $\begingroup$ Thanks! How do you measure empirical reliability in IRT? Is this the same as information? $\endgroup$ – Behacad Sep 7 '13 at 20:07
  • $\begingroup$ Not exactly, it' more a function of how one obtains 'true score' estimates ($\hat{\theta}$) and their associated standard errors, to form the CTT ratio $r_{xx} = T / (T + E)$. So if you compute EAP scores, for example, you can use this information to form the ratio between the variance and $\theta$ and the variance in the standard errors. The mirt package will do this with it's fscores() function, and so will the sirt package (or maybe it's the TAM package....I can't recall, it's the same author for both). $\endgroup$ – philchalmers Sep 7 '13 at 21:22
  • $\begingroup$ @philchalmers,pls take a look question if you can answer it. $\endgroup$ – WhiteGirl Jul 26 '17 at 15:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.