Is there a point to using EFA for scale validation when you can always fit a second-order CFA?

Question

I have been trying to understand the use of Exploratory Factor Analysis for the purpose of scale validation.

Say you have developed a scale to measure construct X, which is supposed to be one unified construct, but your EFA tells you that you actually have 4 factors instead of 1 factor.

Wouldn't you always be able to claim that your construct is valid by fitting a second-order CFA with the EFA factors at level 1 and a single factor at level 2?

And in the latter case, your estimate for the level 2-latent variable is going to equal some weighted sum of the original indicators anyway?

Answer 1

Starting with your latter point first, it's not the case that scores for the higher-order factor would be calculated using some sort of crude weighted sum. Unique/error variance in your indicators for the first-order factors is going to get left behind, and only the shared variance in the first-order factors is going to be captured by the higher-order factor, so only in the (unlikely) case that all observed variance in your indicators was shared variance from the higher-order factor would something like a sum-score work. And even then, the construction of sum scores on the basis of a true latent variable requires an awfully strange measurement model (Rose et al., 2019).

The matter of providing strong evidence of a higher-order factor is, itself, also more difficult than many appreciate, as one must appraise its model adequacy against numerous competitors (e.g., bifactor models, correlated lower-order factor models, single factor models, exploratory structural equation models, etc). A few references/examples to consider in this process are Brunner et al. (2012), Chen et al. (2006), Chen et al. (2012), Credé & Harms (2015), Morin et al. (2016; highly recommended), Mulaik & Quartetti (1997), and Wiesner et al. (2013).

Finally, I will note that in my own personal experience, many researchers are (too) comfortable creating "overall" scores (sums or averages) of ostensibly higher-order/general factors on the basis of correlated factor models without ever actually directly appraising the adequacy of higher-order/bifactor models.

References

Brunner, M., Nagy, G., & Wilhelm, O. (2012). A tutorial on hierarchically structured constructs. Journal of Personality, 80, 796-846.

Chen, F. F., West, S. G., & Sousa, K. H. (2006). A comparison of bifactor and second-order models of quality of life. Multivariate Behavioral Research, 41, 189-225.

Chen, F. F., Hayes, A., Carver, C. S., Laurenceau, J. P., & Zhang, Z. (2012). Modeling general and specific variance in multifaceted constructs: A comparison of the bifactor model to other approaches. Journal of Personality, 80, 219-251.

Credé, M., & Harms, P. D. (2015). 25 years of higher‐order confirmatory factor analysis in the organizational sciences: A critical review and development of reporting recommendations. Journal of Organizational Behavior, 36, 845-872.

Morin, A. J., Arens, A. K., & Marsh, H. W. (2016). A bifactor exploratory structural equation modeling framework for the identification of distinct sources of construct-relevant psychometric multidimensionality. Structural Equation Modeling: A Multidisciplinary Journal, 23, 116-139.

Mulaik, S. A., & Quartetti, D. A. (1997). First order or higher order general factor?. Structural Equation Modeling: A Multidisciplinary Journal, 4, 193-211.

Rose, N., Wagner, W., Mayer, A., & Nagengast, B. (2019). Model-Based Manifest and Latent Composite Scores in Structural Equation Models. Collabra: Psychology, 5.

Wiesner, M., & Schanding, G. T. (2013). Exploratory structural equation modeling, bifactor models, and standard confirmatory factor analysis models: Application to the BASC-2 behavioral and emotional screening system teacher form. Journal of School Psychology, 51, 751-763.