1
$\begingroup$

I have come across a situation in a dataset using a not very used scale, evaluated partly using confirmatory factor analysis (CFA). The CFA has provided support for a number of subscales. In some of them however, factor loadings are fairly low for certain items. When this scale has been used for analysis, full and scale scores have been calculated using the mean of observed scores.

My preferred solution when using the scale myself would have been to do this in a SEM framework, performing a CFA on our data, and using latent scores as predictors for observed outcomes. However, to facilitate comparison with previous studies using observed scores, my colleagues would like us to use a similar rating solution (the mean of observed scores).

Does this make sense? My gut feeling is that there is something off with this solution, especially considering the low factor loadings, but I am not well versed enough in theory/stats to strongly argue this point. Cronbach's alpha is alright-ish for all subscales. Thus, I come here asking for any and all input and ideas!

$\endgroup$

2 Answers 2

3
$\begingroup$

From a measurement perspective, it would likely be preferable to examine the relationships within an SEM in which latent factors represent the scales rather than within a regression or path model that is based on observed sum scores. The reason is that in an SEM with latent variables, the indicators can be properly weighted through estimation of factor loadings, measurement error in the predictors is taken into account through the measurement models, and model (mis)fit can be examined. When using observed sum scores (e.g., in regression), the components (items) are typically given the same weight (which may be incorrect), measurement error in the independent variables (predictors) is not taken into account, and model misfit may not be evident or cannot be examined (regression models are typically saturated).

Perhaps you could do both (observed and latent score analysis) and see how much the results differ? If the differences are strong, perhaps this will convince your colleagues that the SEM approach is likely preferable.

$\endgroup$
1
$\begingroup$

Item Analysis

I would say that what you do here depends on your approach and with each decision comes some pros and cons. First, we must consider what a sum or mean score actually achieves. Here I draw from McNeish and Walsh, 2020 but simplify the explanation a little. The typical way items are summed involves a formula such as below:

$$ y = \text{item}_1 + \text{item}_2 ... \text{item}_k $$

But this comes baked with it the assumption that the items are weighted equally by a constant and have no error in estimation, for example:

$$ y = (1 \times \text{item}_1) + (1 \times \text{item}_2) ... (1 \times \text{item}_k) $$

This also generalizes to means, as you are simply dividing the same weighted sum by the number of items to get an unweighted mean. When it is the case that your items actually behave the same way, a raw sum score or mean score actually does a good job of measuring what you intend (Widaman & Revelle, 2022) and can be included in a statistical analysis without much more investigation or manipulation. In reality, these items are weighted differently based on how well they map onto a latent variable such as the one you describe. Consider if we conducted a CFA and Item 1 ($y_1$) has a near perfect association with the construct but Item 2 ($y_2$) is weakly correlated, such that the standardized loadings are

$$ y_{1} = \tau_1 + (.90 \times \lambda) + \epsilon_{1} \\ y_{2} = \tau_2 + (.10 \times \lambda) + \epsilon_{2} $$

where $\epsilon$ denotes the leftover random noise (error) in estimation and $\tau$ and $\lambda$ represent the intercepts and slopes. Then using these items as a sum or mean score provides an imprecise estimation of what you are trying to measure, which may necessitate different estimation methods.

To SEM or Not to SEM, That is the Question

To that effect and as Christian already noted, using a latent variable based approach with something like a structural equation model probably attacks this problem in the most direct way. Probably the most practical reason for doing this is that the reliability of items is accounted for (aka item errors) and this potentially reduces attenuation bias in your coefficients. The major downside to this approach is sample size. In order to achieve any accuracy you will need to have enough data to generate solutions. You could consider using a Bayes approach if that is an issue, though I would invest some light reading into that topic.

Using something like a vanilla, off-the shelf regression technique isn't inherently wrong, but the traditional assumption is that your predictors are measured with perfect or near perfect precision (Kline, 2023, p.20). If your measure doesn't achieve this (such as what you noted with the low factor loadings), you will lose some accuracy, and so you may need to prune some items in order to achieve some semblance of reliability in estimation. Providing some estimation of the composite reliability (of which I prefer McDonald's $\omega$, see Flora, 2020), will at the minimum be required, with some kind of statement about the uncertainty produced if the reliability isn't super high. I also recently saw an R package that accounts for measurement error in regression estimation that may be useful to read up on (Nab et al., 2021), as it could serve as some kind of compromise in your situation.

References

  • Flora, D. B. (2020). Your coefficient alpha is probably wrong, but which coefficient omega is right? A tutorial on using R to obtain better reliability estimates. Advances in Methods and Practices in Psychological Science, 3(4), 484–501. https://doi.org/10.1177/2515245920951747
  • Kline, R. B. (2023). Principles and practice of structural equation modeling (5th ed.). The Guilford Press.
  • McNeish, D., & Wolf, M. G. (2020). Thinking twice about sum scores. Behavior Research Methods, 52(6), 2287–2305. https://doi.org/10.3758/s13428-020-01398-0
  • Nab, L., Van Smeden, M., Keogh, R. H., & Groenwold, R. H. H. (2021). Mecor: An R package for measurement error correction in linear regression models with a continuous outcome. Computer Methods and Programs in Biomedicine, 208, 106238. https://doi.org/10.1016/j.cmpb.2021.106238
  • Sengewald, M.-A., & Pohl, S. (2019). Compensation and amplification of attenuation bias in causal effect estimates. Psychometrika, 84(2), 589–610. https://doi.org/10.1007/s11336-019-09665-6
  • Widaman, K. F., & Revelle, W. (2022). Thinking thrice about sum scores, and then some more about measurement and analysis. Behavior Research Methods, 55(2), 788–806. https://doi.org/10.3758/s13428-022-01849-w
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.