How to reduce question set of a poorly designed questionnaire?

Question

I have a very lengthy but poorly designed questionnaire, which has a very large number of questions. The questionnaire really needs to be completely rewritten, but prior to that being done, there is an organisational need to use it again in the short term.

Given the length of the questionnaire, it would be helpful to reduce the number of questions. Some of the 'factors' are composed entirely of questions in a Likert score format - the average of these questions is taken to obtain the 'factor' score. It is likely that some of the questions only contribute noise to the score.

Would it seem reasonable to undertake a factor analysis of only the questions lying under each 'factor' and look for a one factor solution, eliminating those questions which don't load to that factor?

Answer 1

@Peter and @ Jonathan have some good ideas. Without repeating, I'll add two more.

1. If you need to shorten your survey and you won't suffer by reducing the sample size you could give different people somewhat different surveys.

2. Reanalyze old data two ways: with and without questions you suspect are less important. Then see if the results of the analysis, and any recommendations which might stem from them, are substantially different when certain questions are excluded.

Answer 2

I've been there before! Legacy surveys: can't live with them, can't live without them...

While I agree with @Peter Flom's approach to look at correlations and reduce the number of highly correlated statements (this is a standard approach to reducing what I call "brick walls" of rating scales in surveys, and has the advantage of being simple enough for most people to understand and buy into your changes), there are a couple of other things I would point out.

1. Survey length impacts responses through "differential mortality", or that some respondents will be more likely to fail to complete than others. Typically, but not always, you will find that happier respondents are more likely to complete a long survey than unhappy respondents. If your main reason to continue using this questionnaire is to provide comparisons to prior data, then shortening the survey may not serve your needs. If you only need to use this one more time, you may be better off leaving it as is for now and focusing your efforts on the complete re-write.
2. Some correlations reflect redundancy, some reflect coincidence: make sure you bring a level of judgment to what statements you choose to eliminate. A standard design for a hotel satisfaction survey will include a question about the customer's overall satisfaction with check-in, and later ask satisfaction with different aspects of the check-in experience like door staff, desk staff, etc. They will all be highly correlated with each other in a typical situation, but each drives different actions in the business, and though they are correlated now, if one aspect of the business reduces its performance, that's something the survey should detect.
3. Consider an organizational solution: You might be able to eliminate a lot of questions just by asking the consumers of the data which different questions actually matter to them. If the survey is GIGO to start with, the statistical approach might just be layering on top of a bad base.

Answer 3

One thing you could do is look at correlations among all the questions and see if any are very high (say, over .9 or some such). For those pairs, you could eliminate one question at random. That would shorten things, at least.

You could also look at Cronbach's alpha for the items you think are under each factor, and eliminate questions that have very low correlations with the others.

Neither of these is going to make the questionnaire much better.... There's always GIGO, and you say the questionnaire is very poor. But they should make it a bit shorter.

Answer 4

I would go for Item Response Theory (IRT). IRT not only lets you check for unidimensionality but also things like Item Information Curves (or Item Response Functions), which might come in handy if you want to reduce the number of items but still want to precisely measure participants broadly on the whole latent scale.

There are a lot of packages in R that fit these kind of models. The ones I have most often used are:

• eRm

• Mokken

I would suggest that you have a look at the Mokken package first, because it includes an automated item selection procedure (aisp function), which automatically builds good scales in an exploratory way (note that "good" is defined here in terms of the so called scalability coefficient H).