McDonald's Omega: Assumptions, Coefficients and Interpretation

Question

After browsing cross validated and several other sources on the web, I still cannot get a grip on McDonald's Omega as a measure of internal consistency. I have a hunch that many fellow social scientists feel similarly insecure about the measure, so I hope to get some clarification on several aspects on this measure:

Assumptions / Prerequisites

While the assumptions for Cronbach's Alpha are commonly discussed (e.g. Cronbach Alpha Assumptions), I haven't managed to get a full picture of the prerequisites for McDonald's Omega. My questions being:

• What are the general assumptions underlying Omega?
• Is there a rule of thumb regarding sample size, or a ratio between variables and observations that should be considered?
• Is Cronbach's Alpha superior to Omega under any circumstances at all?

Coefficients and Interpretation

Secondly, it appears that there still is a great deal of confusion around the different Omega coefficients, perhaps most notably returned by the psych-package in R. For clarification, maybe someone could offer a full interpretation of coefficients in the following example, in ?psych::omega,

library(psych)
#create 9 variables with a hierarchical structure
v9 <- sim.hierarchical()

#find omega 
v9.omega <- omega(v9,digits=2)

> v9.omega$omega.group
        total   general     group
g   0.7984002 0.6857363 0.1126608
F1* 0.7449332 0.6034008 0.1415325
F2* 0.6303512 0.4034189 0.2269323
F3* 0.5022309 0.2460886 0.2561423

> v9.omega$omega.lim
[1] 0.858888

My questions regarding this example:

• How does the interpretation between omega.tot and omega_h (general) differ in this example? Or: What would the correct global measure of internal consistency for the entire measure/questionnaire be?
• What is group telling us?
• When is omega.lim relevant?

In addition: It appears that omega_h (general) is getting the most attention in posts/reports, but these values always strike be as surprisingly low in almost every example I have seen. How come?

Thanks

Answer 1

The topic is old, but the questions interesting, so I would like to include some of available information regarding some questions.

Statistical requirements/assumptions underlying Omega:

Omega and omega hierarchical are based on parameter estimates (i.e., estimates of factor loadings and factor variances) that are derived for a certain CFA model. Hence, two vital statistical requirements need to be fulfilled: (1) Proper interpretation of omega and omega hierarchical requires that the target model fits the empirical data well (2) Parameter estimates need to be precise Brunner, Nagy, Wilhelm, 2012

Rule of thumb regarding sample size, or a ratio between variables and observations that should be considered?

Similarly, the sample size should follow the CFA sample size definition, using preferably simulation methods, as those enabled by R simsem package.

sample size needs to be sufficiently large to obtain trustworthy estimates of model parameters (Yang & Green, 2010).5 In general, a larger sample size is always better, and a sample size of N 200 allows proper estimation of model parameters (e.g., nonnegative variances of subtest-specific factors) under a large variety of conditions (Boomsma & Hoogland, 2001). There is also growing consensus that the required sample size depends on the properties of the model investigated and the data to be analyzed: A higher ratio of measures per factor and higher factor loadings may compensate for smaller sample size (Marsh, Hau, Balla, & Grayson, 1998; Yang & Green, 2010). Thus, methodologists strongly encourage applied researchers to conduct Monte Carlo studies of the target CFA models to determine the required sample size (L. K. Muthén & Muthén, 2002).Brunner, Nagy, Wilhelm, 2012

An interesting reference for this discussion is available on: Brunner M, Nagy G, Wilhelm O. A tutorial on hierarchically structured constructs. J Pers. 2012;80(4):796-846. doi:10.1111/j.1467-6494.2011.00749.x

Answer 2

A useful source that alleviated some of my confusion around this topic was:

McNeish, Daniel. 2018. “Thanks Coefficient Alpha, We’ll Take It from Here.” Psychological Methods 23(3):412–33. doi: 10.1037/met0000144.

One important note is that the Omega-function in the psych (refereed to as Revelle’s omega total. by the paper) package is different to many other implementations.

Answer 3

Interpretation:

Revelle's psych HowTo guide (page ~7) notes that 3 results are typically presented, and these are based on [McDonald, 1999]'s two omega functions:

• Omega hierarchical ($\omega_h$) is an estimate of the general factor saturation of a test.
• Omega hierarchical infinite ($\omega_h$-inf) is the omega for an infinite length test with a structure similar to the observed test.
  • (NB: I interpret this test as the same hypothetical hierarchical structure defined for $\omega_h$ and fitted via a Factor Analysis procedure, ie. E-CFA. In this case the fitting is not limited by a number of iterations, and therefore is likely to maximize the fit. I empirically note it tends to receive a higher coefficient value that $\omega_h$.)
• Omega total ($\omega_t$) is an estimate of the total reliability of a test, including the general factor, group factors (i.e. first-order factors) and item indicators.

Assumptions / Prerequisites:

• In Cronbach's Alpha a unidimensional structure is evaluated for reliability; with predeclared assumptions, such as equal factor loadings (tau-equivalence) and only a single general factor.

• In Omega a fitted hypothetical hierarchical structure of factors is tested for reliability. The user must define which hypothetical hierarchical structures to test for reliability.

psych's Omega implementation seems to support for a comphrensive set of factor structure varieties (e.g. general/first-order, bifactor, hierarchical) which are evaluated after fitting via a factor analysis (FA) modelling procedure (e.g. CFA/SEM, E-CFA/E-SEM, etc.). It seems to have facilities to evaluate and simulate different structure types, report the factor loadings in diagrammatic form and calculate fitness measurements (indices) of the FA models.

If the assumptions of Cronbach's Alpha or the assumptions of the user's hypothetical hierarchical factor model for McDonald's Omega functions are not held, then all will incorrectly report the reliability.

Therefore, confirm the Model Fit Before Proceeding:

To confirm a hierarchical factor model's assumptions is to measure its fitness to the data. Therefore, each hypothetical structure (for Omega) should be evaluated through their Factor Analysis model fitness metrics (see [Hooper, 2008; Hu & Bentler 1999] or answer).

Examples of Factor Analysis model fitness metrics:

• [Hu & Bentler, 1999] (see this answer) identified absolute and relative measure of fit:
  • Absolute: e.g. RMSEA (<.06 / <.08) or SPMR (<.08)
  • Relative: e.g. CFI/TLI (>.95 / >.9)
• Other metrics included in the Python FactorAnalyzer library:
  • Kaiser-Meyer-Olkin Criterion (>.6 / >.8)
  • Bartlett’s Sphericity (null hypothesis) test (Pvalue < critical alpha e.g. <0.05).

If the hypothetical structure model has an adequate fit of the data, then its reliability can be accepted and reported.

Fundamental, Intepretation:

• ($\omega_h$) and ($\omega_h$-inf) are estimates of reliability to a general factor.
• ($\omega_t$) is the estimate for the overall hypothetical hierarchical model.
• Coefficient values are dependable, only if the hypothetical factor analysis model loadings had a good fit.

(This is a rough-guide of how one can apply and interpret Omega, as I understand it.)

References:

• Hooper, D., Coughlan, J., & Mullen, M. R. (2008). Structural equation modeling: Guidelines for determining model fit. Electronic Journal of Business Research Methods, 6(1), 53-60.
• Hu, L., Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.
• McDonald, R.P. (1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum.
• Revelle W. (2023) Using R and the psych package to find ω. Department of Psychology, Northwestern University (June 29, 2023)