I am doing a research validating a new questionnaire, which has 156 items divided up in 12 scales. I have run a factor analysis at scale level, which gives me two nice constructs (consistent with theory).

My tutor however is insisting this is no good as according to her I need to run an exploratory factor analysis at item level. I have done this and found 43 factors (only a couple of values in this huge table have an absolute square value greater than 0.4, which is the value suggested by Field (2005) as being meaningful.

Is it possible that when running such an analysis with a questionnaire with 100+ items, a factor analysis is really not that appropriate?

I have also been reading the PAI manual – PAI Structure chapter of the PAI questionnaire development (pp. 275-289), as this is a questionnaire that has been developed with lots of funds for research and it is now widely used. No factor analysis has been done on the items, but just at scale level. Several subsequent factor analysis carried out by other authors have also just included scales (not items).

I hope to hear some other thoughts on this, ideally with references to study/theories.

  • 2
    $\begingroup$ I dont need an answer on the PAI, which is a well validated psychological questionnaire. My study is on a new questionnaire. My point is that on the PAI validation studies the factor analysis was only done (and several other studies replicated this) at scale level, not item level. This is a link to the PAI sigmaassessmentsystems.com/assessments/pai.asp en.wikipedia.org/wiki/Personality_Assessment_Inventory but my answer is not about the PAI, but another questionnaire, which is a new one. $\endgroup$ – Linda Sep 16 '12 at 17:05
  • $\begingroup$ how have you assigned items to scales without carrying out a factor analysis? $\endgroup$ – richiemorrisroe Sep 16 '12 at 19:36
  • $\begingroup$ It sounds like your "factor analysis at scale level" intend to explore a second-order factor structure. Is this right? In this case, this amounts to ask what best method shall we use to assess 1st- and 2nd-order factor structure? $\endgroup$ – chl Sep 16 '12 at 19:51
  • $\begingroup$ What do you mean by second-order structure? sorry I am not familiar with the distinction. $\endgroup$ – Linda Sep 16 '12 at 20:22
  • $\begingroup$ Yes, items have been assigned to scales based on the DSM-IV, they list clinical symptoms. There are 156 items and they have not been assigned to scales following any factor analysis, but just based on face validity. $\endgroup$ – Linda Sep 17 '12 at 10:07

In this answer I will not differentiate between Factor Analysis (FA) and Principal Component Analysis (PCA), but by default I mean PCA. These two are different. In my environment, when someone says "I do factor analysis" they always do mean PCA, almost never realizing the (subtle) difference.

Analysis on both items and scales can be seen as correct and not interchangeable, as they concern a little different problem. People in psychology usually do item-based FA, and probably that's why your tutor (maybe a little too automatically) asks you do for it. Here are the important differences:

  1. There is much less information in 12 scales than in 156 items - so you were able to discern much more factors (information) from the items. You can set a limit to factor analysis and get only two factors and hope to expect comparable results, but...
  2. ...Factor analysis procedure incorporates a prior belief, that all items are equivalently good. This prior will result in a different bias if some scales in your questionnaire are based on very different amount of items then the others. I know many multiple-scale questionnaires with some scales on 2 or even 1 items, and other on as many as 20. In case of such questionnaires the items present in little scales will bear much more weight upon the FA result than the items present in the large scales. And if there are items shared between the scales, they will automatically get greater impact on the factors.

My advise:

Do as your tutor asks - FA on items is also a good and valid procedure.

I doubt your 43 factors is a valid result of a factor analysis - to me it more sounds like a upper bound.

A proper FA rarely ends in number of factors equal to an upper bound given by the Kaiser criterion (leave all factors with eigenvalue greater than one). The procedure calls you to consider all possible factor sets (honouring the Kaiser criterion and possibly VariMax-rotated) for which you could give a good name/meaning for each and every factor found. I usually end up in a solution with half the factors than the upper bound. And analysing so many sets factors (where first set contains 43 factors) is a hard work, which hardly can be automatized (except maybe for a deep neural network ;-) ).

What works for me the best is starting from the factor analysis with maximum number of factors and working my way down until I find either a set of factors for which I can give a clear meaning or reach a scree criterion (inflection point of the scree plot) that gives a left bound for number of factors.

Timothy Brown - Confirmatory Factor Analysis for Applied Research, page 23:

(...) Despite the fact that EFA is an exploratory or descriptive technique by nature, the decision about the appropriate number of factors should be guided by substantive considerations, in addition to the statistical guidelines discussed below. For instance, the validity of a given factor should be evaluated in part by its interpretability; for example, does a factor revealed by the EFA have substantive importance? A firm theoretical background and previous experience with the variables will strongly foster the interpretability of factors and the evaluation of the overall factor model. Moreover, factors in the solution should be well defined—that is, comprised of several indicators that strongly relate to it. (...)

If you want to test the theory that your questionnaire has exactly two factors - use a Confirmatory Factor Analysis (it is a special case of path analysis AKA Structural Equation Modeling (SEM) ). But that is a different story.

  • $\begingroup$ In my environment, when someone says "I do factor analysis" they always do mean PCA Just to be curious, what is the environment? $\endgroup$ – ttnphns Sep 26 '16 at 7:28
  • $\begingroup$ @ttnphns Poland, professors and assistants in psychology departments of various universities... And handbooks for students too. But don't tell anyone ;-) $\endgroup$ – Adam Ryczkowski Sep 26 '16 at 7:32
  • $\begingroup$ Adam, mine is mostly similar. But I wouldn't argue "they always do mean PCA" when they want FA. Well, perhaps only students or young PhD candidates inexperienced in data analysis. :-) $\endgroup$ – ttnphns Sep 26 '16 at 7:39
  • $\begingroup$ @ttnphns I base my assumption on the fact that SPSS does (or at least did when I last used it) PCA on FA by default. And yes, my sampling is biased as I usually talk to people who actually need a help in statistics. $\endgroup$ – Adam Ryczkowski Sep 26 '16 at 7:42
  • $\begingroup$ @ttnphns but I have at least two scripts/handbooks for students on FA that don't even acknowledge the issue. $\endgroup$ – Adam Ryczkowski Sep 26 '16 at 7:42

I could go on with Adam's two good points about possible differences between the scale-level and item-level factor analysis.

  1. FA on 156 items however calls for a larger sample size than FA on 12 scales.

  2. If the scales had been invented as orthogonal factors in past FA then the present FA can be "successful" only to the extent to which the scales correlate, that is, the extent to which the original FA was inadequate or the current test application is incomparable (different population etc) with the original one. But if the scales had been invented as oblique factors then the doing FA on these scales now can be seen as an attempt to perform second-order (1,2) factor analysis skipping the re-doing of the first-order one.

  3. If the scales had been invented not based on item inter-correlation paradigm (such as FA) but differently (such as for example based specifically on the correlations with the target criterion) then the scale-level FA is likely to be incomparable with the item-level FA.

  4. Summation of many item scores into a scale score can be seen - from the reliability standpoint - as cancelling-out random error variability meddling with the implied "true feature". Scale-level FA appears then as if the analysis of error-free data. On the other hand, item-level FA will not cancel but greet those same variabilities as potentially systematic ones: it could join some lesser common factors (split-off from the main ones) or remain in the unique ("error") factors, - depending on your FA solution. So, the two approaches are conceptually somewhat different, mirroring the distinction between reliability and validity notions. Alpha-factor analysis method could be done on items if you want to cling more to the reliability standpoint.

  5. Summation items into scale is tied with the question how much adequate it is in comparison with, for example, more "fine" estimates of a trait such as factor scores. Scale level FA might be quite crude (and ultimately loosing comparability with the item-level FA) if the plain summation is a bad estimation of the trait values.


I think Item Factor Analysis and Subscale Factor Analysis could be seen as two steps of a general procedure to evaluate measurements.

  • 3
    $\begingroup$ Welcome to the site, @Guest. This seems like the beginning to a useful answer for the OP. Would you care to elaborate it a little? $\endgroup$ – gung - Reinstate Monica Nov 22 '14 at 23:46
  • $\begingroup$ Clarification: I voted to delete because this remained un-elaborated on since two years. It is safe to assume this answer will never be elaborated. $\endgroup$ – amoeba says Reinstate Monica Sep 26 '16 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.