We are developing a questionnaire for evaluation of burden of disease after certain medical conditions. The questions are developed partly from semistructured interviews. My statistician argues for using both exploratory and confirmatory FA for item reduction etc. The questionnaire is large; it contains questions in what I think will turn out to be 10-13 domains, each containing 6-30 items.
I'm curious as to how to handle the fact that we're using a control group. We have a group of 2000 patients and 500 in a health control group. As I understand FA will be an excellent way of extracting factors, deciding of number of factors to retain and performing rotation. We would proably use maximum likelihood, then parallel analysis followed by a proper rotation.
In my (novice) understanding of FA, this would be a typical way of reducing a questionnaire in for example psychological tests, where one could discuss latent variables, factors etc.

  1. Are these methods really useful when it comes to item-reduction in questionnaires asking for symptoms and comparing to a healthy control group?
  2. Would one first perform the FA on answers from the un-healthy group to see what questions "falls out" per se and then compare those specific questions with the healthy control group or would one instead perform the same analysis on the entire study population at once?
  3. Is FA, at least exploratory, entirely wrong in this context? We have already created empirical domains e.g. "Sleep" which contains questions like "How often do you wake up at night?" but other questions like "Have you had negative thoughts about your body" could fit within both the sexual domain as well as the physical and the psychological. In my simple mind the first example would not need an analysis to find out under which domain it belongs, but the second example would. On the other hand, maybe exploratory FA would be an ojective way of finding out whether the domains are "correct" or not?

Please downvote if these questions not really fit the rules on CV, so that I can delete it. Otherwise, please guide me through these steps. My background is, rather obviously, not in statistics.


1 Answer 1


Having worked on any number of projects where factor analysis (or PCA) was used as a dimension reducing step, I would answer your first question with a resounding yes, these methods are quite useful in making a sprawling set of survey items more manageable or tractable for further analysis.

That said, I've known teams to literally spend weeks deciding on the "right" set of factors. As far as I'm concerned, this is a waste of time. Get in and get out...that's the best way to deal with exploratory PCA since, as a consequence of having done a PCA, you are not permanently obliged to retain and use these dimensions in interpreting the results of your analysis. It simply means that PCAs are a very useful and ultimately disposable intermediate step -- e.g., they are particularly useful in running any additional multivariate analyses such as clustering or segmentation. You would want to retain and focus on the granularity of the specific question items for later insight and interpretation in favor of the factor dimensions which, since they are composite variables summarizing every input feature can deliver counterintuitive findings.

Of course, if your goal is to create new, scaled dimensions in the psychometric sense of that term, then this exploratory work would naturally evolve into a later confirmatory phase with the contingent tests of reliability, discriminant and convergent validity of the resulting scaled domains. That's a different and more rigorous objective from the typical applied use of PCAs.

Wrt question #2, you would want to do the work of PCA on the full sample, breaking out the control group from others as part of the post-hoc profiling of the results. Since your data is unbalanced (control group << the rest of the sample) one consideration is whether or not to weight. If you do decide on using weighting, do it after you have developed a PCA not while developing the dimensions. This is a general "best" practice or rule of thumb for which I do not have a good reference.

Question #3 concerns the validity of judgment versus statistically driven domains. In my experience, it's not all that common for "judgment" driven factors to be confirmed (with a high degree of precision) by a statistically driven PCA but it definitely does occur. That said, whether one does one approach or the other is entirely a subjective team decision -- both can be motivated or justified and neither one would be "wrong" in any strategically important sense of that word. Usually decisions like these are a function of the culture in which the work is being done and the most senior people usually determine that, not the analyst crunching the numbers. If there's an upside to exploratory PCA it lies in the possibility for "serendipitous" findings and insights that the a priori, theoretically driven domains or factors might have missed.

  • $\begingroup$ Thank you very much DJohnson! I will try to digest all this information and then maybe return with additional questions. Just one semanthical question: Since most of the questions already belongs in an a priori hypothesized factor (like questions about sleep disturbances are presumed to belong in the Sleep domain), could one argue that we perform exploratory FA to confirm our hypothesis, although the actual confirmatory FA will be performed at a later stage? $\endgroup$
    – Johan_A_M
    Nov 9, 2015 at 13:42
  • $\begingroup$ No question but that you can use exploratory PCA as a validatory step. Based on my experience, you will get back approximate representations of your original, judgment-driven domains, not precise recoveries. $\endgroup$
    – user78229
    Nov 9, 2015 at 18:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.