Is subjective weighting acceptable to create composites for correlation analysis?

Question

Following on from my related questions on weighting scales at multiple levels and differences between indexes and composites, I have two composite scores, X and Y.

Both have been constructed through subjective weighting. For each, I code each of my 10 likert items on a 0 to 5 scale (0 = not important to 5 = absolute importance). I then sum the item scores to form a scale score.

As you can see, I am using a very simple formula which is not based on any statistical tests but on common logic of relative importance of the responses for each likert scale responses.

I am hesitant to use statistical tests to work out the weights for two main reasons.

• Firstly, I want to be in control of the weights and therefore don't want abstract tests to determine the weights.
• Secondly, my model is for people who want a practical model rather than a complex statistical model.

Questions

• In this case, how do I demonstrate that my model is valid?
• Do I have to convert all scores into Z scores before I add them up?

Answer 1

Describing your system for forming a composite

You have weighted a set of items using a standard system (i.e., equal distance between numbers for categories [0,1,2,3,4,5], the same scale for each item; items scores then summed to form a scale score).

I would not call the above system "subjective". It is probably the most common system for scoring psychology and social science multi-item scales when items use the same response scale.

I imagine that you are contrasting such a scoring system with one based on a factor analysis or a related procedure.

Validity of your composite

You are adopting a standard scoring system, and there are good reasons why this system is so common.

• It's very easy to communicate to others how the scoring system works, and therefore can readily be applied in multiple contexts.
• It is not specific to a given sample (in contrast to weightings derived from a factor analysis; although such weightings could be fixed in one sample and applied in others).
• Many scales are designed so that each item is designed to measure the scale and therefore summing over these items is designed to measure the construct.
• Because each item is on the same scale (and the number of scale points is relatively small), the standard deviation for items tend to be fairly similar, and thus the contribution of the item to the scale total tends to be fairly similar.

Nonetheless, such a scoring system is predicated on the idea that each item is a good measure of the underlying construct. The broader issue of validity relates to whether the scale you have created is a valid measure of whatever it is meant to be a measure of. There are a wide range of procedures that people use to establish the validity of a given scale both in general, and for a specific sample.

• Factor analysis and reliability analysis are two obvious ways to assess the internal structure of a scale or set of scales.
• Correlating the scale with other measures is another strategy.

Much more could be said about validity and scale construction (see here for some references).

Whether to convert items to Z-scores

Standardising items first before creating a scale mean or sum is one simple way of ensuring that each item has equivalent "importance" in forming a composite. The problem of unequal importance is more of a problem when combining component variables that are on very different scales (e.g., height in mm, weight in kg). There are also issues in comparability across studies when adopting a sample specific z-score approach.

In your case, whether you convert each item to a z-score first before summing items or whether you just sum items, my guess is that the two variables will correlate very highly (perhaps greater than .95, but you can check this). In general the simple sum (or mean) of raw scores is preferable from a comparability perspective. It also communicates how the mean relates to the underlying scale (e.g., a mean of 4.2 on a 1 to 5 scale where 5 means very satisfied indicates that the sample is generally satisfied, whereas the sum of z-scores does not).