My question is to create an index to measure a manager's earnings forecast ability. I use earnings forecast accuracy (dollars difference) and the earnings forecast horizon (days difference) to capture the ability. However, those two variables condition on each other. How should I weight them and create index? The smaller dollar difference and the longer horizon will reflect higher forecast ability.
$\begingroup$
$\endgroup$
1
-
$\begingroup$ The predict error shows their ability, which is the difference of predict earnings and actual earnings. However, the horizon might affect the predict error. All I do now is to rank them into deciles and then take the average. Do you think are they comparable? $\endgroup$– user10757Commented Apr 20, 2012 at 0:27
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
If there really is an underlying ability then it should certainly be the case that the items you use to measure it correlate with one another (which is what I assume 'condition on' means here). Indeed, most explicit measurement models, e.g. factor analysis, assume that you want the model such that defines the underlying ability is the quantity that, if it were to be known, there would be no remaining correlation to explain. So if you are thinking of weighting these quantities, you are implicitly thinking of applying such a measurement model because factor analysis 'scores' are (often) essentially weighted averages of the model's items.
In any case, whether or not you actually want to fit such a model it will be helpful to think what sort of transformation of your items would make it best behaved if you did. You have two difference measures and, conditional on the unobserved ability of your manager, these look like they will have rather different variances. For example, earnings projections in 10s and in 1000s should have errors of similar orders of magnitude and long and short term deals should have differences on a long and short time scale, both regardless of actual ability. So you might be better transforming them to make different managers comparable. (This is the equivalent in a model of asking for reasonably constant item variance conditional on ability).
For example, you might consider working with a chi-square type measure that rescales each item by dividing the difference between managers' predictions and their outcomes by their predictions. At that point you can decide whether you want to simply take an average or run a factor analysis on it to get weights.
answered Apr 17, 2012 at 22:32
-
$\begingroup$ Conjugate, thank you for your answer! So I need to standardize my variables first and then average them? How about rank order them and weighted by ranking order scales? or use decile? Do you have any book or good website to recommend? I only have a very basic stat background...Thanks again. $\endgroup$– WendyCommented Apr 18, 2012 at 0:11
-
$\begingroup$ If you like the answer please feel free to vote it up. $\endgroup$ Commented Apr 18, 2012 at 12:18
-
$\begingroup$ There is no unique answer to what you 'need' to do. Two considerations may help though: 1) ranking or otherwise chopping up your items will always lose information, so it had better at least improve interpretation to make up for that. 2) Think about what counts as a good result for each item and make sure the measure reflects that, e.g. following my middle paragraph, if a manager A predicts 1000 and gets 750 and manager B predicts 600 and gets 400 are they as good as each other, or is A better? Which transformation will reflect your intuitions and which not? $\endgroup$ Commented Apr 18, 2012 at 12:27
-
$\begingroup$ This online book seems reasonable. Simpler versions will be found in most psychology textbooks that cover scale construction. $\endgroup$ Commented Apr 18, 2012 at 12:31