Fixed-sum question

Question

I have a set of 10 items each with a weight between 0-1. A person can rate each of these 10 items with a score between 0-100, but is given, say, only 500 points to allocate among these 10 items The person does not know the weight of each item and the total score will be the weighted average.

A person could rate each item as 50 and another person could rate the top 5 items with the highest weights with all 100's. The weights are such that the difference in total scores received by rating all items with 50, and only the top-5 highest weighted items with 100s and the remaining 5 items with 0, is not significant given that the spread in the weights (all between 0-1) is not high.

Given that a person can randomly rate all the items with 50's and still receive a total score very close to the highest possible score obtained by rating the top-5 weighted items with all 100's, is problematic.

Are there any existing alternative approaches/methods/algorithms to solve such a problem so that a person who rates all items 50 will get a score far away from the maximum possible score?

Answer 1

Ultimately we'd need to see the weighting algorithm itself to answer this question; otherwise we wouldn't know if a particular input transform invalidated the weighting algorithm producing the final score. That said:

First thing to consider before changing the outcomes of that weighting scheme: Is that particular use of variable weighting and limited point allocation an established practice in your particular research domain? If it is then you should be very careful changing the variable inputs directly. You could invalidate the methodology altogether. At minimum it will change the scaling of that variable and have implications for comparing your results to other results using the original methodology.

If so and your intent is to remove people who didn't establish any preference across the 10 items then I'd create a new dichotomous/binary (dummy) variable identifying them and run your tests filtering on that or as a covariate. Perhaps you can retrain your weights filtering those people in and out?

Addressing the fixed sum issue
If it's not a common practice and instead a field experiment by you or the survey designers that you're now trying to work around you have a couple of options beyond creation of new variable identifying people who just split points evenly, per the above point.

1. Transform within items - one example would be to recenter the range on 0, running -50 to 50, then take their absolute value or square the values. Again, such methods probably invalidate the scoring algorithm.
2. Transform across items - take the average of all 10 scores and then transform each individual variable by re-scoring it to the difference from the mean. This becomes a measure of relative preference/sentiment/etc across the 10 items, and again could invalidate the methodology.
3. Do #2 to create a new single measure in addition to the weighted output score, and then scale the output score by this new measure. This may or may not be a good idea depending on what the scoring algorithm itself is doing - you have to evaluate that closely.

A final option of course is to revisit the scoring algorithm itself if it's not part of an accepted methodology in your domain and alter so that it best suits your needs, using some of the techniques above.

Answer 2

One easy fix would be to subtract 50 -- or, more generally, (# of points allocated)/(# of items) -- from each rating before multiplying by the secret weight. That wouldn't change the differences between people's total scores, but it would make it easy to spot people who are scoring better (or worse) than chance because their scores would be positive (or negative).

Answer 3

With the weights given in your comment of $0.63,0.63,0.63,0.60,0.56,0.56,0.53,0.50,0.50,0.44,0.38,0.38,​0.38$, with a total of $650$ to allocate and $0-100$ per item, the maximum possible score is as you say $387.5$ while giving $50$ per item gives $336$ and the minimum possible score is $284.5$.

So the indifferent scorer gets a score near the centre of the range (in this particular example, exactly between the maximum and the minimum, but that need not be true with other weights). Not surprisingly, the indifferent scorer's score is equal to the expected score of somebody allocating points at random.

The person giving $50$ to each item is unlikely to have the highest score even when there are only three people playing and the other two are giving points at random. If the others are using good judgement in terms of which items are likely to have higher weights then it becomes even less likely the equal-point player will win.