Sum of rating scores vs estimated factor scores?

Question

I'd be interested to receive suggestions about when to use "factor scores" over plain sum of scores when constructing scales. I.e. "Refined" over "non-refined" methods of scoring a factor. From DiStefano et al. (2009; pdf), emphasis added:

There are two main classes of factor score computation methods: refined and non-refined. Non-refined methods are relatively simple, cumulative procedures to provide information about individuals’ placement on the factor distribution. The simplicity lends itself to some attractive features, that is, non-refined methods are both easy to compute and easy to interpret. Refined computation methods create factor scores using more sophisticated and technical approaches. They are more exact and complex than non-refined methods and provide estimates that are standardized scores.

To my mind, if the goal is to create a scale that can be used across studies and settings, then a simple sum or average score of all scale items makes sense. But let's say that the goal is to evaluate treatment effects of a program and the important contrast is within sample—treatment vs control group. Is there any reason why we might prefer factor scores to scale sums or averages?

To be concrete about the alternatives, take this simple example:

library(lavaan)
library(devtools)

# read in data from gist ======================================================
# gist is at https://gist.github.com/ericpgreen/7091485
# this creates data frame mydata
  gist <- "https://gist.github.com/ericpgreen/7091485/raw/f4daec526bd69557874035b3c175b39cf6395408/simord.R"
  source_url(gist, sha1="da165a61f147592e6a25cf2f0dcaa85027605290")
  head(mydata)
# v1 v2 v3 v4 v5 v6 v7 v8 v9
# 1  3  4  3  4  3  3  4  4  3
# 2  2  1  2  2  4  3  2  1  3
# 3  1  3  4  4  4  2  1  2  2
# 4  1  2  1  2  1  2  1  3  2
# 5  3  3  4  4  1  1  2  4  1
# 6  2  2  2  2  2  2  1  1  1

# refined and non-refined factor scores =======================================
# http://pareonline.net/pdf/v14n20.pdf

# non-refined -----------------------------------------------------------------
  mydata$sumScore <- rowSums(mydata[, 1:9])
      mydata$avgScore <- rowSums(mydata[, 1:9])/9
  hist(mydata$avgScore)

# refined ---------------------------------------------------------------------
  model <- '
            tot =~ v1 + v2 + v3 + v4 + v5 + v6 + v7 + v8 + v9
           '
  fit <- sem(model, data = mydata, meanstructure = TRUE,
             missing = "pairwise", estimator = "WLSMV")
  factorScore <- predict(fit)
  hist(factorScore[,1])

Answer 1

I've been wrestling with this idea myself in some current projects. I think you need to ask yourself what is being estimated here. If a one-factor model fits, then the factor scores estimate the latent factor. The straight sum or mean of your manifest variables estimates something else, unless every observation loads equally on the factor, and the uniquenesses are also the same. And that something else is probably not a quantity of great theoretical interest.

So if a one-factor model fits, you are probably well advised to use the factor scores. I take your point about comparability across studies, but within a particular study, I think the factor scores have a lot going for them.

Where it gets interesting is when a one-factor model does not fit, either because a two-factor model applies (or higher), or because the covariance structure is more complicated than a factor model predicts. To me, the question is then whether the straight total of the variables refers to anything real. This is particularly true if the data have more than one dimension. In practice, what often happens is that you have a bunch of related variables (items on a survey, perhaps), with one or two of them being way different from the others. You can say, "to Hell with this", and take the average of everything, regardless of what it means. Or you can go with the factor scores. If you fit a one-factor model, what will typically happen, is that the factor analysis will downweight the less useful variables (or at least, those variables that really belong on a second factor score). In effect, it spots them as belonging to a different dimension and ignores them.

So I believe that the factor score can sort of prune the data to give something more uni-dimensional than you started with. But I don't have a reference for this, and I'm still trying to figure out in my own work if I like this approach. To me, the big danger is overfitting when you plough the scores into another model with the same data. The scores are already the answer to an optimization question, so where does that leave the rest of the analysis? I hate to think.

But at the end of the day, does a sum or total of variables actually make sense if something like a one-factor model does not apply?

A lot of these questions would not arise if people designed better scales to start with.

Answer 2

Summing or averaging items loaded by the common factor is a traditional way to reckon the construst score (the construct representing tha factor). It is a simplest version of the "coarse method" of computing factor scores; the method's main point stands in using factor loadings as score weights. While refined methods to compute scores use specially estimated score coefficients (calculated from the loadings) as the weights.

This answer does not universally "suggest about when to use [refined] factor scores over plain sum of item scores", which is a vast domain, but focuses on showing some concrete obvious implications going with preferring one way of reckoning the construct over the other way.

Consider a simple situation with some factor $F$ and two items loaded by it. According to Footnote 1 here explaining how regressional factor scores are computed, factor score coefficients $b_1$ and $b_2$ to compute factor scores of $F$ come from

$s_1=b_1r_{11}+b_2r_{12}$,

$s_2=b_1r_{12}+b_2r_{22}$,

where $s_1$ and $s_2$ are the correlations between the factor and the items - the factor loadings; $r_{12}$ is the correlation between the items. The $b$ coefficients are what distinguish factor scores from simple, unweighted sum of the item scores. For, when you compute just the sum (or mean) you deliberately set both $b$s to be equal. While in "refined" factor scores the $b$s are got from the above equations and are not equal usually.

For simplicity, and because factor analysis is often performed on correlations let us take the $r$s as correlations, not covariances. Then $r_{11}$ and $r_{22}$ are unit and can be omitted. Then,

$b_1 = \frac{s_2r_{12}-s_1}{r_{12}^2-1}$,

$b_2 = \frac{s_1r_{12}-s_2}{r_{12}^2-1}$,

hence $b_1-b_2= -\frac{(r_{12}+1)(s_1-s_2)}{r_{12}^2-1}.$

We are interested in how this potential inequality between the $b$s is dependent on the inequality among the loadings $s$s and the correlation $r_{12}$. The function $b_1-b_2$ is shown below on the surface plot and also on a heatmap plot.

Clearly, as the loadings are equal ($s_1-s_2=0$) the $b$ coefficients are also equal, always. As $s_1-s_2$ grows, $b_1-b_2$ grows in response, and grows the more rapidly the greater is $r_{12}$.

So, if two items are loaded by their factor about equally you may safely set their weights equal, i.e. compute simple sum, - because the $b$ weights (which determine regressional factor scores) are about equal too. You do not depart far from factor scores (a).

But consider two different loadings, say, $s_1=.70$ and $s_2=.45$, the difference is $.25$. If you choose to simply sum their scores given by a respondent the degree how much awry is your decision relative to the estimated factor score depends on how strongly the items correlate with each other. If they correlate not very strongly, your bias is not too pronounced (b). But if they correlate really strongly, the bias is strong too, so simple sum won't do (c). Interpreting the reason in the three situations:

c. If they correlate strongly, the weaker loaded item is a junior duplicate of the other one. What's the reason to count that weaker indicator/symptom in the presense of its stronger substitute? No much reason. And factor scores adjust for that (while simple summation doesn't). Note that in a multifactor questionnaire the "weaker loaded item" is often another factor's item, loaded higher there; while in the present factor this item gets restrained, as we see now, in computation of factor scores, - and that serves it right.

b. But if items, while loaded as before unequally, do not correlate that strongly, then they are different indicators/symptoms to us. And could be counted "twice", i.e. just summed. In this case, factor scores try to respect the weaker item to the extent its loading still allows, for it being a different embodiment of the factor.

a. Two items can also be counted twice, i.e. just summed, whenever they have similar, sufficiently high, loadings by the factor, whatever correlation between these items. (Factor scores add more weight to both items when they correlate not too tight, however the weights are equal.) It seems not unreasonable that we usually tolerate or admit quite duplicate items if they are all strongly loaded. If you don't like this (sometimes you may want to) you are ever free to eliminate duplicates from the factor manually.

So, in computation of (refined) factor scores (by the regression method at least) there apparent are "get along / push out" intrigues among the variables constituting the construct, in their influence on the scores. Equally strong indicators tolerate each other, as unequally strong not strongly correlated ones do, too. "Shutting up" occurs of a weaker indicator strongly correlated with stronger indicators. Simple addition/averaging doesn't have that "push out a weak duplicate" intrigue.

Please see also this answer which warns that factor theoretically is rather an "essence inside" than a gross collection or heap of "its" indicative phenomena. Therefore blindly summing up items - taking neither their loadings nor their correlations in mind - is potentially problematic. On the other hand, factor, as scored, can be but some kind of a sum of its items, and so everything is about a better conception of the weights in the sum.

---

Let us also glance at deficiency of coarse or summation method more generally and abstractly.

In the beginning of the answer I've said that obtaining a construct score via plain summing/averaging is a particular case of coarse method of factor score reckoning whereby score coefficients $b$s are replaced by factor loadings $a$s (when the loadings enter dichotomized as 1 (loaded) and 0 (unloaded) we get exactly that simple summing or averaging of items).

Let $\hat F_i$ be a respondent $i$ factor score (estimate of value) and $F_i$ be his true factor value (ever unknown). We also know that each of items $X1$ and $X2$ loaded by the common factor (with loadings $a1$ and $a2$) consist of that common factor $F$ plus the unique factor $U$ (we assume the latter comprising specific factor S and error term e). So, in reckoning factor scores as packages do via $b$s we have

$\hat F_i = b1X1_i+b2X2_i = b1(F_i+U1_i)+b2(F_i+U2_i) = (b1+b2)F_i+b1U1_i+b2U2_i$.

If $b1U1_i+b2U2_i$ happens to be close to zero $\hat F_i$ and $F_i$ are equivalent. Unless unique factors $U$s are altogether absent (or unless we known their values, which we don't) we can never provide $\hat F$ scores reflecting $F$ values precisely. We could, however, contrive the two $b$ coefficients in such a way that $\text{var}[b1U1_i+b2U2_i]$ is possibly minimal across respondents; then $\hat F$ will strongly correlate with $F$. One method or another, by estimating score coefficients $b$s from loadings $a$s and values $X$ we can make $\hat F$ scores be quite representative of $F$.

But look at the "coarse method" - where loadings $a$s themselves are admitted in place of $b$s to the above approximation of $F$ by $\hat F$:

$\hat F_i = a1X1_i+a2X2_i= ~...~ =(a1+a2)F_i+a1U1_i+a2U2_i$.

What we see here is weighting of unique factors by those same coefficients that are the degree how variables are weighted by the common factor. Above, $b$s were computed with the help of $a$s, true, but they weren't $a$s themselves; and now $a$'s themselves came to weight as they are - to weight what they relate to not. This is the crudity we commit when using "coarse method" of factor score computation, including plain summation/averaging of items as its specific variant.