How to reduce number of items using factor analysis, internal consistency, and item response theory in conjunction?

Question

I am in the process of empirically developing a questionnaire and I will be using arbitrary numbers in this example to illustrate. For context, I am developing a psychological questionnaire aimed at assessing thought patterns commonly identified in individuals who have anxiety disorders. An item could look like "I need to check the oven repeatedly because I can't be sure its off".

I have 20 questions (5-point Likert) which may be comprised of one or two factors (note that in reality I have closer to 200 questions, comprised of 10 scales, and each scale may be comprised of two factors). I am willing to erase about half the items, leaving 10 questions on one of two factors.

I am familiar with exploratory factor analysis (EFA), internal consistency (Cronbach's alpha), and item characteristic curves in item response theory (IRT). I can see how I would use any single of these methods to determine which items are the "worse" within any single scale. I appreciate that each method also answers different questions, although they may lead to similar results and I am not sure what "question" is most important.

Before we start, lets make sure I know what I am doing with each of these methods individually.

• Using EFA, I would identify the number of factors, and remove the items that load the least (lets say <.30) on their respective factor or that cross-load substantially across factors.

• Using internal consistency, I would remove items that have the worse "alpha if item deleted". I could do so assuming one factor in my scale, or do it after an initial EFA to identify the number of factors and subsequently run my alpha for each factor.

• Using IRT, I would remove items that do not assess the factor of interest along their (5 Likert) response options. I would be eyeballing item characteristic curves. I would basically be looking for a line on a 45 degree angle going from option 1 on the Likert scale all the way up to 5 along the latent score. I could do so assuming one factor, or do it after an initial
  EFA to identify number of factors, and subsequently run the curves for each factor.

I am unsure which of these methods to use in order to best identify which items are the "worst". I use worst in a broad sense such that the item would be of detriment to the measure, either in terms of reliability or validity, both of which are equally important to me. Presumably I can use them in conjunction, but I am not sure how.

If I was to go ahead with what I know now and give it my best shot I would do the following:

1. Do an EFA to identify number of factors. Also delete items with bad loadings on their respective factors, since I don't want items that load badly regardless of how they would do in other analyses.
2. Do IRT and remove bad items judged by that analysis as well, if any remain from the EFA.
3. Simply report Cronbach's Alpha and don't use that metric as a means to delete items.

Any general guidelines would be greatly appreciated!

Here is also a list of specific questions that you can perhaps answer:

1. What is the practical difference between removing items based on factor loadings and removing items based on Chronbach's alpha (assuming you use the same factor layout for both analyses)?

2. Which should I do first? Assuming I do EFA and IRT with one factor, and both identify different items that should be removed, which analysis should have priority?

I am not hard set on doing all of these analyses, although I will report Chronbach's alpha regardless. I feel like doing just IRT would leave something missing, and likewise for just EFA.

Answer 1

I don't have any citations, but here's what I'd suggest:

Zeroth: If at all possible, split the data into a training and test set.

First do EFA. Look at various solutions to see which ones make sense, based on your knowledge of the questions. You'd have to do this before Cronbach's alpha, or you won't know which items go into which factor. (Running alpha on ALL the items is probably not a good idea).

Next, run alpha and delete items that have much poorer correlations than the others in each factor. I wouldn't set an arbitrary cutoff, I'd look for ones that were much lower than the others. See if deleting those makes sense.

Finally, choose items with a variety of "difficulty" levels from IRT.

Then, if possible, redo this on the test set, but without doing any exploring. That is, see how well the result found on the training set works on the test set.

Answer 2

All three of your suggested criteria actually could be performed in IRT, more specifically multidimensional IRT. If your sample size is fairly large would probably be a consistent way to go about it for each subscale. In this way you could get the benefits of IRT for modelling item independently (using nominal models for some items, generalized partial credit or graded for others, or if possible even set up rating scales to help interpret polytomous items in a more parsimonious way).

MIRT is conceptually equivalent to item-level factor analysis and therefore has a linear EFA equivalent relationship for dichotomous and polytomous items. I'm not sure if I would buy into the <.3 criteria for dropping items though, since it really depends on the context and factor structure. Small loadings/slopes don't provide as much information about the intercept locations, but may still be useful since they can offer a wider and less peaked information function across levels of $\theta$. Some applications in CAT make use of these types of items early on as well since they give a wider band of information early on in the test.

Dropping items based on the Cronbach criteria is roughly the same as dropping items that give a better marginal/empirical reliability in IRT, so if the software you are using supports these statistics then you could follow the same strategy without leaving the IRT paradigm. I'd be more inclined to check the information functions however to see if removing an item severely affects the measurement at various $\theta$ levels (related to where the intercepts are). Relative information plots are useful here as well.

You could attempt to remove items that don't conform to the unidimensional requirements of most IRT software, but I wouldn't necessarily recommend this if it affects the theoretical representation of the constructs at hand. In empirical applications it's usually better to try and make our models fit our theory, not the other way around. Also, this is where the bifactor/two-tier models tend to be appropriate since you would like to include all possible items while accounting for multidimensionality in a systematic and theoretically desirable way.