Sum of rating scores vs estimated factor scores?

I'd be interested to receive suggestions about when to use "factor scores" over plain sum of scores when constructing scales. I.e. "Refined" over "non-refined" methods of scoring a factor. From DiStefano et al. (2009; pdf), emphasis added:

There are two main classes of factor score computation methods: refined and non-refined. Non-refined methods are relatively simple, cumulative procedures to provide information about individuals’ placement on the factor distribution. The simplicity lends itself to some attractive features, that is, non-refined methods are both easy to compute and easy to interpret. Refined computation methods create factor scores using more sophisticated and technical approaches. They are more exact and complex than non-refined methods and provide estimates that are standardized scores.

To my mind, if the goal is to create a scale that can be used across studies and settings, then a simple sum or average score of all scale items makes sense. But let's say that the goal is to evaluate treatment effects of a program and the important contrast is within sample—treatment vs control group. Is there any reason why we might prefer factor scores to scale sums or averages?

To be concrete about the alternatives, take this simple example:

library(lavaan)
library(devtools)

# read in data from gist ======================================================
# gist is at https://gist.github.com/ericpgreen/7091485
# this creates data frame mydata
gist <- "https://gist.github.com/ericpgreen/7091485/raw/f4daec526bd69557874035b3c175b39cf6395408/simord.R"
source_url(gist, sha1="da165a61f147592e6a25cf2f0dcaa85027605290")
# v1 v2 v3 v4 v5 v6 v7 v8 v9
# 1  3  4  3  4  3  3  4  4  3
# 2  2  1  2  2  4  3  2  1  3
# 3  1  3  4  4  4  2  1  2  2
# 4  1  2  1  2  1  2  1  3  2
# 5  3  3  4  4  1  1  2  4  1
# 6  2  2  2  2  2  2  1  1  1

# refined and non-refined factor scores =======================================
# http://pareonline.net/pdf/v14n20.pdf

# non-refined -----------------------------------------------------------------
mydata$sumScore <- rowSums(mydata[, 1:9]) mydata$avgScore <- rowSums(mydata[, 1:9])/9
hist(mydata$avgScore) # refined --------------------------------------------------------------------- model <- ' tot =~ v1 + v2 + v3 + v4 + v5 + v6 + v7 + v8 + v9 ' fit <- sem(model, data = mydata, meanstructure = TRUE, missing = "pairwise", estimator = "WLSMV") factorScore <- predict(fit) hist(factorScore[,1])  • I've removed "interventions" from the title to make the question sound more general and because interventions possibly do not have unique, specific bearing on the distinction between the two kinds of construct reckoning. Please, you are welcome to roll back my edit if you don't agree. – ttnphns Jan 19 '16 at 11:00 • They are more exact This added emphasis should not distract us from the fact that even factor scores are inevitably inexact ("underdetermined"). – ttnphns Jan 19 '16 at 11:03 • See also this similar question: stats.stackexchange.com/q/31967/3277. – ttnphns Jan 19 '16 at 11:16 • I think "interventions" is relevant as a special use case, but it does not need to be in the title. I've highlighted the key issue in the question. As for the emphasis on "more exact", I was curious to get thoughts about this point given the observation you make about factor scores being indeterminate. Thanks for the links to other questions. – Eric Green Jan 19 '16 at 13:30 • "more exact". Among linearly computed factor scores the regression method is most "exact" in the sense "most correlated with the unknown true factor values". So yes, more exact (within linear algebraic approach), but not totally exact. – ttnphns Jan 19 '16 at 14:30 2 Answers I've been wrestling with this idea myself in some current projects. I think you need to ask yourself what is being estimated here. If a one-factor model fits, then the factor scores estimate the latent factor. The straight sum or mean of your manifest variables estimates something else, unless every observation loads equally on the factor, and the uniquenesses are also the same. And that something else is probably not a quantity of great theoretical interest. So if a one-factor model fits, you are probably well advised to use the factor scores. I take your point about comparability across studies, but within a particular study, I think the factor scores have a lot going for them. Where it gets interesting is when a one-factor model does not fit, either because a two-factor model applies (or higher), or because the covariance structure is more complicated than a factor model predicts. To me, the question is then whether the straight total of the variables refers to anything real. This is particularly true if the data have more than one dimension. In practice, what often happens is that you have a bunch of related variables (items on a survey, perhaps), with one or two of them being way different from the others. You can say, "to Hell with this", and take the average of everything, regardless of what it means. Or you can go with the factor scores. If you fit a one-factor model, what will typically happen, is that the factor analysis will downweight the less useful variables (or at least, those variables that really belong on a second factor score). In effect, it spots them as belonging to a different dimension and ignores them. So I believe that the factor score can sort of prune the data to give something more uni-dimensional than you started with. But I don't have a reference for this, and I'm still trying to figure out in my own work if I like this approach. To me, the big danger is overfitting when you plough the scores into another model with the same data. The scores are already the answer to an optimization question, so where does that leave the rest of the analysis? I hate to think. But at the end of the day, does a sum or total of variables actually make sense if something like a one-factor model does not apply? A lot of these questions would not arise if people designed better scales to start with. • I appreciate your comments, @Placidia. You bring some clarity while reminding us of the bigger mess! I think this is an interesting point to consider: "If the factor model fits, then the factor scores estimate the latent factor. The straight sum or mean of your manifest variables estimates something else, unless every observation loads equally on the factor, and the uniquenesses are also the same. And that something else is probably not a quantity of great theoretical interest." – Eric Green Jan 18 '16 at 21:30 • +1 for a very thoughtful answer. A couple of thoughts to add: 1) regarding comparability across studies, it is important to recognize that unlike component loadings--which can change quite a bit in response to variables included/excluded from the model--common factor loadings are parameter estimates. Subsequently, they should replicate (within sampling error) from study to study, and thus, so too should the factor scores. 2) If you are concerned about using factor scores, you might look at indexes of determinacy and how well your factor score correlations mirror the latent correlations... – jsakaluk Jan 19 '16 at 5:24 • ...as I think this is a strategy discussed in the DiStefanno et. al. paper to evaluate whether factor scores can be "trusted". And finally 3) if your goal, as Placidia describes, is to analyze something that is largely unidimensional, you might consider a bifactor analysis approach, which as I understand it, first extracts a common factor onto which every variable loads, and then subsequent orthogonal factors are extracted for subsets of variables, which ostensibly reflect the most important distinguishable factors, beyond the common dimension tying all the variables together. – jsakaluk Jan 19 '16 at 5:27 • Placidia, in the last edit of your answer, you repeatedly constrain yourself by the expression one-factor model. I just wonder why. Are you saying that in a 2-factor model factor scores don't estimate the latent factor anymore? Why so? And also, how do you define "one-factor model" in the context of a being developed questionnaire (the Q's likely context): is that the questionnaire is single-factor/scale or that each included item is counted belonging strictly to one factor/scale? Please, will you mind making it more clear? – ttnphns Jan 25 '16 at 18:24 • I wanted to avoid potential misunderstanding. If you believe in a two-factor model, presumably using summary totals would be off the table. You need two summaries for two dimensions in the data. I wanted to clarify that my answer was about choosing between the summary statistic and the factor score of the one-factor model. I claim that the one-factor score can be useful even if the model is false. @jsakaluk's suggestion of fitting a multi-factor model and choosing the first factor is also possible, and might be better in some cases. – Placidia Jan 25 '16 at 19:01 Summing or averaging items loaded by the common factor is a traditional way to reckon the construst score (the construct representing tha factor). It is a simplest version of the "coarse method" of computing factor scores; the method's main point stands in using factor loadings as score weights. While refined methods to compute scores use specially estimated score coefficients (calculated from the loadings) as the weights. This answer does not universally "suggest about when to use [refined] factor scores over plain sum of item scores", which is a vast domain, but focuses on showing some concrete obvious implications going with preferring one way of reckoning the construct over the other way. Consider a simple situation with some factor$F$and two items loaded by it. According to Footnote 1 here explaining how regressional factor scores are computed, factor score coefficients$b_1$and$b_2$to compute factor scores of$F$come from$s_1=b_1r_{11}+b_2r_{12}$,$s_2=b_1r_{12}+b_2r_{22}$, where$s_1$and$s_2$are the correlations between the factor and the items - the factor loadings;$r_{12}$is the correlation between the items. The$b$coefficients are what distinguish factor scores from simple, unweighted sum of the item scores. For, when you compute just the sum (or mean) you deliberately set both$b$s to be equal. While in "refined" factor scores the$b$s are got from the above equations and are not equal usually. For simplicity, and because factor analysis is often performed on correlations let us take the$r$s as correlations, not covariances. Then$r_{11}$and$r_{22}$are unit and can be omitted. Then,$b_1 = \frac{s_2r_{12}-s_1}{r_{12}^2-1}$,$b_2 = \frac{s_1r_{12}-s_2}{r_{12}^2-1}$, hence$b_1-b_2= -\frac{(r_{12}+1)(s_1-s_2)}{r_{12}^2-1}.$We are interested in how this potential inequality between the$b$s is dependent on the inequality among the loadings$s$s and the correlation$r_{12}$. The function$b_1-b_2$is shown below on the surface plot and also on a heatmap plot. Clearly, as the loadings are equal ($s_1-s_2=0$) the$b$coefficients are also equal, always. As$s_1-s_2$grows,$b_1-b_2$grows in response, and grows the more rapidly the greater is$r_{12}$. So, if two items are loaded by their factor about equally you may safely set their weights equal, i.e. compute simple sum, - because the$b$weights (which determine regressional factor scores) are about equal too. You do not depart far from factor scores (a). But consider two different loadings, say,$s_1=.70$and$s_2=.45$, the difference is$.25$. If you choose to simply sum their scores given by a respondent the degree how much awry is your decision relative to the estimated factor score depends on how strongly the items correlate with each other. If they correlate not very strongly, your bias is not too pronounced (b). But if they correlate really strongly, the bias is strong too, so simple sum won't do (c). Interpreting the reason in the three situations: c. If they correlate strongly, the weaker loaded item is a junior duplicate of the other one. What's the reason to count that weaker indicator/symptom in the presense of its stronger substitute? No much reason. And factor scores adjust for that (while simple summation doesn't). Note that in a multifactor questionnaire the "weaker loaded item" is often another factor's item, loaded higher there; while in the present factor this item gets restrained, as we see now, in computation of factor scores, - and that serves it right. b. But if items, while loaded as before unequally, do not correlate that strongly, then they are different indicators/symptoms to us. And could be counted "twice", i.e. just summed. In this case, factor scores try to respect the weaker item to the extent its loading still allows, for it being a different embodiment of the factor. a. Two items can also be counted twice, i.e. just summed, whenever they have similar, sufficiently high, loadings by the factor, whatever correlation between these items. (Factor scores add more weight to both items when they correlate not too tight, however the weights are equal.) It seems not unreasonable that we usually tolerate or admit quite duplicate items if they are all strongly loaded. If you don't like this (sometimes you may want to) you are ever free to eliminate duplicates from the factor manually. So, in computation of (refined) factor scores (by the regression method at least) there apparent are "get along / push out" intrigues among the variables constituting the construct, in their influence on the scores. Equally strong indicators tolerate each other, as unequally strong not strongly correlated ones do, too. "Shutting up" occurs of a weaker indicator strongly correlated with stronger indicators. Simple addition/averaging doesn't have that "push out a weak duplicate" intrigue. Please see also this answer which warns that factor theoretically is rather an "essence inside" than a gross collection or heap of "its" indicative phenomena. Therefore blindly summing up items - taking neither their loadings nor their correlations in mind - is potentially problematic. On the other hand, factor, as scored, can be but some kind of a sum of its items, and so everything is about a better conception of the weights in the sum. Let us also glance at deficiency of coarse or summation method more generally and abstractly. In the beginning of the answer I've said that obtaining a construct score via plain summing/averaging is a particular case of coarse method of factor score reckoning whereby score coefficients$b$s are replaced by factor loadings$a$s (when the loadings enter dichotomized as 1 (loaded) and 0 (unloaded) we get exactly that simple summing or averaging of items). Let$\hat F_i$be a respondent$i$factor score (estimate of value) and$F_i$be his true factor value (ever unknown). We also know that each of items$X1$and$X2$loaded by the common factor (with loadings$a1$and$a2$) consist of that common factor$F$plus the unique factor$U$(we assume the latter comprising specific factor S and error term e). So, in reckoning factor scores as packages do via$b$s we have$\hat F_i = b1X1_i+b2X2_i = b1(F_i+U1_i)+b2(F_i+U2_i) = (b1+b2)F_i+b1U1_i+b2U2_i$. If$b1U1_i+b2U2_i$happens to be close to zero$\hat F_i$and$F_i$are equivalent. Unless unique factors$U$s are altogether absent (or unless we known their values, which we don't) we can never provide$\hat F$scores reflecting$F$values precisely. We could, however, contrive the two$b$coefficients in such a way that$\text{var}[b1U1_i+b2U2_i]$is possibly minimal across respondents; then$\hat F$will strongly correlate with$F$. One method or another, by estimating score coefficients$b$s from loadings$a$s and values$X$we can make$\hat F$scores be quite representative of$F$. But look at the "coarse method" - where loadings$a$s themselves are admitted in place of$b$s to the above approximation of$F$by$\hat F$:$\hat F_i = a1X1_i+a2X2_i= ~...~ =(a1+a2)F_i+a1U1_i+a2U2_i$. What we see here is weighting of unique factors by those same coefficients that are the degree how variables are weighted by the common factor. Above,$b$s were computed with the help of$a$s, true, but they weren't$a$s themselves; and now$a\$'s themselves came to weight as they are - to weight what they relate to not. This is the crudity we commit when using "coarse method" of factor score computation, including plain summation/averaging of items as its specific variant.

• Thanks, @ttnphns, for the helpful response. It makes sense to me that items with approximately equal loadings could just be summed (a). Unfortunately, I don't think I've ever encountered a situation in my work where, when using an existing scale that is supposedly unidimensional, I find that the items have equal loadings. – Eric Green Jan 19 '16 at 13:40
• So I was particularly interested in your explanation of a situation where loadings differ and the suggestion to examine the inter-item correlations. I'm interested to know if you have any rules of thumb for "strong" (c) / "not strong" correlations (b) or "sufficiently high" loadings in (a). – Eric Green Jan 19 '16 at 13:40
• Finally, I'll note that the backdrop of this question is an overwhelming disciplinary norm (at least in psychology) to use "validated" scales that require simple sums (averages) even when administering the scale to new non-normed population. Often the goal is cross-sample comparisons (even when unwarranted), which makes simple sums a common approach. – Eric Green Jan 19 '16 at 13:43
• Intervention studies are an interesting use case in my mind because the comparison of interest is within sample. It would seem to me that we care more about the treatment effect size than either groups "raw" score on the measure—especially when using the scale outside of the population used to develop/norm the scale. If factor scores are "better" in some situations, then it seems worth tossing the simple approach in favor of one that makes more conceptual sense knowing that ultimately we just want to look at treatment effect sizes. – Eric Green Jan 19 '16 at 13:45
• (cont.) Use "validated" scales doesn't itself require necessarily simple sums: if validation was good (representative large sample, good correlations, correct number of factors, nice fit, etc.) the computed factor scores (their coefficients) can be taken as the normative weights to be used in new populations. In this respect, I can't see any advantage in the simple sum. – ttnphns Jan 19 '16 at 14:26