1
$\begingroup$

So, we were reviewing a manuscript where the research group was developing an anxiety survey for upper level biology students. The survey had 25 items; was administered via paper and online formats (N = 200-250 each time) and reliability was > 0.8 with either format. Secondly, they got 4 components using PCA, loadings were fairly high (> 0.7) and subscale reliability was pretty high too.

Based on their high reliability and loadings, the researchers were contemplating on whether items needed to be excluded from the next administration of the survey. From what they had stated, there were no items with cross loadings, no items that consistently showed up without loadings and their 4 components were meaningful. So, why would high reliability, high loadings, a 25 item survey and a decent item:case ratio bring up the question of dropping any items at all? Is there justification for that decision?

$\endgroup$
2
$\begingroup$

Usually the reasoning goes the other way, namely low reliability is a reason to look for additional items. High reliability does not in itself provide a compelling reason to remove items but it means you can do it without worrying about negative consequences. It doesn't mean that you “need” to do anything per se but it makes sense to point the high reliability out in this context.

As @GottfriedHelms noted the main reason for doing this would be convenience/ease of administration, reducing the time needed to fill in the questionnaire and making it easier to integrate in longer survey instruments. In a typical scale building effort, you would generate many more items than you need so maybe they never intended to keep the longer scale. Obviously, it's a little odd not to mention this explicitly but it's still a very good reason.

Note that PCA (and Cronbach $\alpha$, if that's what they use to estimate reliability) is generally not recommended for scale building. Results are often not too different but factor analysis makes more sense in principle.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

I think, that the “ancient” argumentation with computing complexity is obsolete today with the fast computers. However, it might be of interest to reduce the number of items to reduce the burden of the interviewer and of the examined person. For instance, we had a research job to find a short test which is able to indicate development of school- and social skills of 6 to 14 year old children. In the pretest phase we had a collection of about 200 items to be asked - and this cannot be done with young children by the teachers on a regular basis. So one of our goals was to find the optimal subset in the sense of having few items but which still models all factors of development. The final test had then, if I recall correctly, about 40 items and was implemented in the half-year rhythm in the schools.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I do understand that, but why would high reliability/high loadings be a justification for reducing the number of items? The reason was not cited as reducing respondent fatigue or any kind of burden on the interviewer. Just curious as to why when the survey is already resulting in meaningful results, high reliability and loadings would one want to reduce the number of items. Is there a statistical justification for it? $\endgroup$ – user28687 Aug 6 '13 at 5:30
  • $\begingroup$ @user: I for my part cannot think of a statistical or mathematical reason. But my experience is a bit limited, let's wait for more correspondents. $\endgroup$ – Gottfried Helms Aug 6 '13 at 5:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.