Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have a set of 10 items each with a weight between 0-1. A person can rate each of these 10 items with a score between 0-100, but is given, say, only 500 points to allocate among these 10 items The person does not know the weight of each item and the total score will be the weighted average.

A person could rate each item as 50 and another person could rate the top 5 items with the highest weights with all 100's. The weights are such that the difference in total scores received by rating all items with 50, and only the top-5 highest weighted items with 100s and the remaining 5 items with 0, is not significant given that the spread in the weights (all between 0-1) is not high.

Given that a person can randomly rate all the items with 50's and still receive a total score very close to the highest possible score obtained by rating the top-5 weighted items with all 100's, is problematic.

Are there any existing alternative approaches/methods/algorithms to solve such a problem so that a person who rates all items 50 will get a score far away from the maximum possible score?

share|improve this question
1  
Your question appears to be more psychology than statistics. It might be more statistical if you could perhaps operationalize what you mean by 'fraudulent', since such scores are going to be subjective. Could you clarify more precisely what you want to avoid and what you want to achieve? –  Glen_b Oct 13 '12 at 0:12

2 Answers 2

Ultimately we'd need to see the weighting algorithm itself to answer this question; otherwise we wouldn't know if a particular input transform invalidated the weighting algorithm producing the final score. That said:

First thing to consider before changing the outcomes of that weighting scheme: Is that particular use of variable weighting and limited point allocation an established practice in your particular research domain? If it is then you should be very careful changing the variable inputs directly. You could invalidate the methodology altogether. At minimum it will change the scaling of that variable and have implications for comparing your results to other results using the original methodology.

If so and your intent is to remove people who didn't establish any preference across the 10 items then I'd create a new dichotomous/binary (dummy) variable identifying them and run your tests filtering on that or as a covariate. Perhaps you can retrain your weights filtering those people in and out?

Addressing the fixed sum issue
If it's not a common practice and instead a field experiment by you or the survey designers that you're now trying to work around you have a couple of options beyond creation of new variable identifying people who just split points evenly, per the above point.

  1. Transform within items - one example would be to recenter the range on 0, running -50 to 50, then take their absolute value or square the values. Again, such methods probably invalidate the scoring algorithm.
  2. Transform across items - take the average of all 10 scores and then transform each individual variable by re-scoring it to the difference from the mean. This becomes a measure of relative preference/sentiment/etc across the 10 items, and again could invalidate the methodology.
  3. Do #2 to create a new single measure in addition to the weighted output score, and then scale the output score by this new measure. This may or may not be a good idea depending on what the scoring algorithm itself is doing - you have to evaluate that closely.

A final option of course is to revisit the scoring algorithm itself if it's not part of an accepted methodology in your domain and alter so that it best suits your needs, using some of the techniques above.

share|improve this answer
    
and I kinda wonder if the weighting algorithm itself is behaving properly if it produces those results - sounds like it's doing a cross-item evaluation and dropping items that don't get any points allocated to them. See if giving 8 items alternating scores of 62 and 63 produces a similar score to all 50s or 5 100s. –  tabSF Oct 13 '12 at 17:42
    
Here are the weights for the 13 items with each weight between 0-1. 0.63,0.63,0.63,0.60,0.56,0.56,0.53,0.50,0.50,0.44,0.38,0.38,0.38. If we now assign 50/100 score to each item and calculate the total score, it will be 336. Note that the user only has 650 points total in the bucket, but can assign 0-100 for any item as long as there are points remaining in the bucket. If the user assigns 100/100 for each of the top-6 weighted items and 50/100 for the 7th most weighted item, the total score will be 387.5. My problem is that the spread between 336 and 387.5 is not big enough to detect fraud –  Sunil Kosuri Oct 13 '12 at 21:46
    
What I mean by fraud is that a user that follows the strategy of assigning 50 to each of the 13 items will likely always end up with the highest score because it would be very difficult for genuine users who are rating the items correctly to beat this 336 score in most cases. –  Sunil Kosuri Oct 13 '12 at 21:49

One easy fix would be to subtract 50 -- or, more generally, (# of points allocated)/(# of items) -- from each rating before multiplying by the secret weight. That wouldn't change the differences between people's total scores, but it would make it easy to spot people who are scoring better (or worse) than chance because their scores would be positive (or negative).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.