Factor analysis of questionnaires composed of Likert items

Question

I used to analyse items from a psychometric point of view. But now I am trying to analyse other types of questions on motivation and other topics. These questions are all on Likert scales. My initial thought was to use factor analysis, because the questions are hypothesised to reflect some underlying dimensions.

• But is factor analysis appropriate?
• Is it necessary to validate each question regarding its dimensionality ?
• Is there a problem with performing factor analysis on likert items?
• Are there any good papers and methods on how to conduct factor analysis on Likert and other categorical items?

Answer 1

From what I've seen so far, FA is used for attitude items as it is for other kind of rating scales. The problem arising from the metric used (that is, "are Likert scales really to be treated as numeric scales?" is a long-standing debate, but providing you check for the bell-shaped response distribution you may handle them as continuous measurements, otherwise check for non-linear FA models or optimal scaling) may be handled by polytmomous IRT models, like the Graded Response, Rating Scale, or Partial Credit Model. The latter two may be used as a rough check of whether the threshold distances, as used in Likert-type items, are a characteristic of the response format (RSM) or of the particular item (PCM).

Regarding your second point, it is known, for example, that response distributions in attitude or health surveys differ from one country to the other (e.g. chinese people tend to highlight 'extreme' response patterns compared to those coming from western countries, see e.g. Song, X.-Y. (2007) Analysis of multisample structural equation models with applications to Quality of Life data, in Handbook of Latent Variable and Related Models, Lee, S.-Y. (Ed.), pp 279-302, North-Holland). Some methods to handle such situation off the top of my head:

• use of log-linear models (marginal approach) to highlight strong between-groups imbalance at the item level (coefficients are then interpreted as relative risks instead of odds);
• the multi-sample SEM method from Song cited above (Don't know if they do further work on that approach, though).

Now, the point is that most of these approaches focus at the item level (ceiling/floor effect, decreased reliability, bad item fit statistics, etc.), but when one is interested in how people deviate from what would be expected from an ideal set of observers/respondents, I think we must focus on person fit indices instead.

Such $\chi^2$ statistics are readily available for IRT models, like INFIT or OUTFIT mean square, but generally they apply on the whole questionnaire. Moreover, since estimation of items parameters rely in part on persons parameters (e.g., in the marginal likelihood framework, we assume a gaussian distribution), the presence of outlying individuals may lead to potentially biased estimates and poor model fit.

As proposed by Eid and Zickar (2007), combining a latent class model (to isolate group of respondents, e.g. those always answering on the extreme categories vs. the others) and an IRT model (to estimate item parameters and persons locations on the latent trait in both groups) appears a nice solution. Other modeling strategies are described in their paper (e.g. HYBRID model, see also Holden and Book, 2009).

Likewise, unfolding models may be used to cope with response style, which is defined as a consistent and content-independent pattern of response category (e.g. tendency to agree with all statements). In the social sciences or psychological literature, this is know as Extreme Response Style (ERS). References (1–3) may be useful to get an idea on how it manifests and how it may be measured.

Here is a short list of papers that may help to progress on this subject:

1. Hamilton, D.L. (1968). Personality attributes associated with extreme response style. Psychological Bulletin, 69(3): 192–203.
2. Greanleaf, E.A. (1992). Measuring extreme response style. Public Opinion Quaterly, 56(3): 328-351.
3. de Jong, M.G., Steenkamp, J.-B.E.M., Fox, J.-P., and Baumgartner, H. (2008). Using Item Response Theory to Measure Extreme Response Style in Marketing Research: A Global Investigation. Journal of marketing research, 45(1): 104-115.
4. Morren, M., Gelissen, J., and Vermunt, J.K. (2009). Dealing with extreme response style in cross-cultural research: A restricted latent class factor analysis approach
5. Moors, G. (2003). Diagnosing Response Style Behavior by Means of a Latent-Class Factor Approach. Socio-Demographic Correlates of Gender Role Attitudes and Perceptions of Ethnic Discrimination Reexamined. Quality & Quantity, 37(3), 277-302.
6. de Jong, M.G. Steenkamp J.B., Fox, J.-P., and Baumgartner, H. (2008). Item Response Theory to Measure Extreme Response Style in Marketing Research: A Global Investigation. Journal of Marketing Research, 45(1), 104-115.
7. Javaras, K.N. and Ripley, B.D. (2007). An “Unfolding” Latent Variable Model for Likert Attitude Data. JASA, 102(478): 454-463.
8. slides from Moustaki, Knott and Mavridis, Methods for detecting outliers in latent variable models
9. Eid, M. and Zickar, M.J. (2007). Detecting response styles and faking in personality and organizational assessments by Mixed Rasch Models. In von Davier, M. and Carstensen, C.H. (Eds.), Multivariate and Mixture Distribution Rasch Models, pp. 255–270, Springer.
10. Holden, R.R. and Book, A.S. (2009). Using hybrid Rasch-latent class modeling to improve the detection of fakers on a personality inventory. Personality and Individual Differences, 47(3): 185-190.

Answer 2

Exploratory factor analysis (EFA) is appropriate (psychometrically and otherwise) for examining the extent to which one may explain correlations among multiple items by inferring the common influence of (an) unmeasured (i.e., latent) factor(s). If this is not your specific intent, consider alternative analyses, e.g.:

• General linear modeling (e.g., multiple regression, canonical correlation, or (M)AN(C)OVA)
• Confirmatory factor analysis (CFA) or latent trait/class/profile analyses
• Structural equation (SEM) / partial least squares modeling

Dimensionality is the first issue EFA can address. You can examine the eigenvalues of the covariance matrix (as by producing a scree plot via EFA) and conduct a parallel analysis to resolve the dimensionality of your measures. (See also some great advice and alternative suggestions from William Revelle.) You should do this carefully before extracting a limited number of factors and rotating them in EFA, or before fitting a model with a specific number of latent factors using CFA, SEM, or the like. If a parallel analysis indicates multidimensionality, but your general (first) factor vastly outweighs all others (i.e., has by far the largest eigenvalue / explains the majority of variance in your measures), consider bifactor analysis ^{(Gibbons & Hedeker, 1992; Reise, Moore, & Haviland, 2010)}.

Many problems arise in EFA and latent factor modeling of Likert scale ratings. Likert scales produce ordinal (i.e., categorical, polytomous, ordered) data, not continuous data. Factor analysis generally assumes any raw data input are continuous, and people often conduct factor analyses of matrices of Pearson product-moment correlations, which are only appropriate for continuous data. Here's a quote from Reise and colleagues ⁽²⁰¹⁰⁾:

Ordinary confirmatory factor analytic techniques do not apply to dichotomous or polytomous data ^{(Byrne, 2006)}. Instead, special estimation procedures are required ^{(Wirth & Edwards, 2007)}. There basically are three options for working with polytomous item response data. The first is to compute a polychoric matrix and then apply standard factor analytic methods ^{(see Knol & Berger, 1991)}. A second option is to use full-information item factor analysis ^{(Gibbons & Hedeker, 1992)}. The third is to use limited information estimation procedures designed specifically for ordered data such as weighted least squares with mean and variance adjustment ^{(MPLUS; Muthén & Muthén, 2009)}.

I would recommend combining both the first and third approaches (i.e., use diagonally weighted least squares estimation on a polychoric correlation matrix), based on Wang and Cunningham's ⁽²⁰⁰⁵⁾ discussion of the problems with typical alternatives:

When confirmatory factor analysis was conducted with nonnormal ordinal data using maximum likelihood and based on Pearson product-moment correlations, the downward parameter estimates produced in this study were consistent with Olsson's ⁽¹⁹⁷⁹⁾ findings. In other words, the magnitude of nonnormality in the observed ordinal variables is a major determinant of the accuracy of parameter estimates.

The results also support the findings of Babakus, et al. ⁽¹⁹⁸⁷⁾. When maximum likelihood estimation is used with a polychoric correlation input matrix in confirmatory factor analyses, the solutions tend to result in unacceptable and therefore significant chi-square values together with poor fit statistics.

The question remains as to whether researchers should use weighted least squares or diagonally weighted least squares estimators in estimating structural equation models with nonnormal categorical data. Neither weighted least squares nor diagonally weighted least squares estimation makes assumptions about the nature of the distribution of the variables and both methods produce asymptotically valid results. Nevertheless, because weighted least squares estimation is based on fourth-order moments, this approach frequently leads to practical problems and is very computationally demanding. This means that weighted least squares estimation may lack robustness when used to evaluate models of medium, i.e., with 10 indicators, to large size and small to moderate sample sizes.

It isn't clear to me whether the same concern with weighted least squares estimation applies to DWLS estimation; regardless, the authors recommend that estimator. In case you don't have the means already:

• R ^{(R Core Team, 2012)} is free. You'll need an old version (e.g., 2.15.2) for these packages:
  • The psych package ^{(Revelle, 2013)} contains the polychoric function.
    • The fa.parallel function can help identify the number of factors to extract.
  • The lavaan package ^{(Rosseel, 2012)} offers DWLS estimation for latent variable analysis.
  • The semTools package contains the efaUnrotate, orthRotate, and oblqRotate functions.
  • The mirt package ^{(Chalmers, 2012)} offers promising alternatives using item response theory.

I imagine Mplus ^{(Muthén & Muthén, 1998-2011)} would work too, but the free demo version won't accommodate more than six measurements, and the licensed version isn't cheap. It may be worth it if you can afford it though; people love Mplus, and the Muthéns' customer service via their forums is incredible!

As stated above, DWLS estimation overcomes the problem of normality assumption violations (both univariate and multivariate), which is a very common problem, and almost ubiquitous in Likert scale rating data. However, it's not necessarily a pragmatically consequential problem; most methods aren't too sensitive to (heavily biased by) small violations (cf. Is normality testing 'essentially useless'?). @chl's answer to this question raises more important, excellent points and suggestions regarding problems with extreme response style too; definitely an issue with Likert scale ratings and other subjective data.

---

^{References
· Babakus, E., Ferguson, J. C. E., & Jöreskog, K. G. (1987). The sensitivity of confirmatory maximum likelihood factor analysis to violations of measurement scale and distributional assumptions. Journal of Marketing Research, 24, 222–228.
· Byrne, B. M. (2006). Structural Equation Modeling with EQS. Mahwah, NJ: Lawrence Erlbaum.
· Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. Retrieved from http://www.jstatsoft.org/v48/i06/.
· Gibbons, R. D., & Hedeker, D. R. (1992). Full-information item bi-factor analysis. Psychometrika, 57, 423–436.
· Knol, D. L., & Berger, M. P. F. (1991). Empirical comparison between factor analysis and multidimensional item response models. Multivariate Behavioral Research, 26, 457–477.
· Muthén, L. K., & Muthén, B. O. (1998-2011). Mplus user's guide (6th ed.). Los Angeles, CA: Muthén & Muthén.
· Muthén, L. K., & Muthén, B. O. (2009). Mplus (Version 4.00). [Computer software]. Los Angeles, CA: Author. URL: http://www.statmodel.com.
· Olsson, U. (1979). Maximum likelihood estimates for the polychoric correlation coefficient. Psychometrika, 44, 443–460.
· R Core Team. (2012). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, URL: http://www.R-project.org/.
· Reise, S. P., Moore, T. M., & Haviland, M. G. (2010). Bifactor models and rotations: Exploring the extent to which multidimensional data yield univocal scale scores. Journal of Personality Assessment, 92(6), 544–559. Retrieved from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2981404/.
· Revelle, W. (2013). psych: Procedures for Personality and Psychological Research. Northwestern University, Evanston, Illinois, USA. Retrieved from http://CRAN.R-project.org/package=psych. Version = 1.3.2.
· Rosseel, Y. (2012). lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software, 48(2), 1–36. Retrieved from http://www.jstatsoft.org/v48/i02/.
· Wang, W. C., & Cunningham, E. G. (2005). Comparison of alternative estimation methods in confirmatory factor analyses of the General Health Questionnaire. Psychological Reports, 97, 3–10.
· Wirth, R. J., & Edwards, M. C. (2007). Item factor analysis: Current approaches and future directions. Psychological Methods, 12, 58–79. Retrieved from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3162326/.}

Answer 3

Just a short note that you might want to look at polychoric correlation with factor analysis rather than the traditional correlation/covariance matrix.

http://www.john-uebersax.com/stat/sem.htm