How to create index from correlated variables?

Question

My question is to create an index to measure a manager's earnings forecast ability. I use earnings forecast accuracy (dollars difference) and the earnings forecast horizon (days difference) to capture the ability. However, those two variables condition on each other. How should I weight them and create index? The smaller dollar difference and the longer horizon will reflect higher forecast ability.

Answer 1

If there really is an underlying ability then it should certainly be the case that the items you use to measure it correlate with one another (which is what I assume 'condition on' means here). Indeed, most explicit measurement models, e.g. factor analysis, assume that you want the model such that defines the underlying ability is the quantity that, if it were to be known, there would be no remaining correlation to explain. So if you are thinking of weighting these quantities, you are implicitly thinking of applying such a measurement model because factor analysis 'scores' are (often) essentially weighted averages of the model's items.

In any case, whether or not you actually want to fit such a model it will be helpful to think what sort of transformation of your items would make it best behaved if you did. You have two difference measures and, conditional on the unobserved ability of your manager, these look like they will have rather different variances. For example, earnings projections in 10s and in 1000s should have errors of similar orders of magnitude and long and short term deals should have differences on a long and short time scale, both regardless of actual ability. So you might be better transforming them to make different managers comparable. (This is the equivalent in a model of asking for reasonably constant item variance conditional on ability).

For example, you might consider working with a chi-square type measure that rescales each item by dividing the difference between managers' predictions and their outcomes by their predictions. At that point you can decide whether you want to simply take an average or run a factor analysis on it to get weights.