0
$\begingroup$

Scenario:

Consider a statement, e.g. "This movie is an action" Then let people vote on that. 1-5 where 1 is "Not action", 3 is "Some action" and 5 is "Pure action".

Question

How does one determine how accurate the the statement from the voting result?

1000 votes where 900 is a 5 is pretty accurate, but 1000 votes with 500 1's and 500 5's is not very accurate. Also 3 votes and all a 5 is not so accurate.

Basically determine the certainty how much an object is of a specific category from uservotes.

I'm having some problem explaining my idea, but ask any questions and I'll try to clarify

Improve this question
$\endgroup$
3
  • $\begingroup$ Do you have a 'ground truth'/answer key/etc? For example, if 900/1000 people vote the same way, is it possible that they're all wrong (e.g., you're showing them an optical illusion or perhaps they're all just dolts)? $\endgroup$
    – Matt Krause
    Commented Apr 11, 2012 at 9:22
  • $\begingroup$ Not really. Basically I'm trying to let users decide what an object is. Ofcourse I can never be 100% sure that its correct, 900 people can be dolts. But I should be able to say that "It's blah percent chance that this is an ..." $\endgroup$
    – Artog
    Commented Apr 11, 2012 at 9:25
  • $\begingroup$ There's a similar thread that's gotten some answers:stats.stackexchange.com/questions/26054/… $\endgroup$
    – Matt Krause
    Commented Apr 11, 2012 at 10:36

1 Answer 1

Reset to default
3
$\begingroup$

I agree with the comment from Matt that the related thread is the best one to get the answer to the question you want to be asking (http://stats.stackexchange.com/questions/26054/are-observations-consistent-with-each-other).

However, to explain why the linked thread is appropriate, I'll address the question you've actually asked. There is no way to determine the accuracy of a rating in this scenario. Accuracy is only meaningful in relation to a definitive source of truth.

If you were a maker of kitchen scales, you could purchase a lump of metal that is carefully crafted and calibrated and guaranteed to weigh exactly 500 grams, and use that to test if your scales accurately report the weight as 500 grams. But, this requires believing that the calibrated weight is a source of absolute truth. If you have a group of random scales that all report different weights, then you have no idea which are accurate, only which mostly agree with each other and which don't. That's an example with weights, which is a single-dimensional scalar measurement; with more subjective topics like describing movies, there are much more ways for people to disagree. They can interpret the movie through distinct cultural lenses that manifest as clusters of similar ratings, they can disagree about the meaning of the words in the description, they can count some parts of the movie as more or less important in determining their overall description, and so on.

So you can see that there is no accuracy; there is only agreement, and many different kinds and levels of disagreement that may manifest themselves in different ways. Some kinds of disagreement might imply that the description category should be re-written (disagreement about meanings) and some kinds might imply membership in distinct groups who might benefit from views of information that's most relevant and likely to agree with their views.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.