Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Let's say, we have simple "yes/no" question that we want to know answer to. And there are N people "voting" for correct answer. Every voter has a history - list of 1's and 0's, showing whether they were right or wrong about this kind of questions in the past. If we assume history as a binomial distribution, we can find voters' mean performance on such questions, their variation, CI and any other kind of confidence metrics.

Basically, my question is: how to incorporate confidence information into voting system?

For example, if we consider only mean performance of each voter, then we can construct simple weighted voting system:

$$result = sign(\sum_{v \in voters}\mu_v \times (-1)^{1-vote})$$

That is, we can just sum voters' weights multiplied either by $+1$ (for "yes") or by $-1$ (for "no"). It makes sense: if voter 1 has average of correct answers equal to $.9$, and voter 2 has only $.8$, than, probably, 1st person's vote should be considered as more important. On other hand, if 1st person have answered only 10 questions of this kind, and 2nd person have answered 1000 such questions, we are much more confident about 2nd person's skill level than about those of the 1st - it's just possible that 1st person was lucky, and after 10 relatively successful answers he will continue with much worse results.

So, more precise question may sound like this: is there statistical metric that incorporates both - strength and confidence about some parameter?

share|improve this question
up vote 4 down vote accepted

You should consider the expertise of a voter as a latent variable of your system. You may then be able to solve your problem with bayesian inference. A representation as graphical model could be like this :


Let's denote the variables $A$ for the true answer, $V_i$ for the vote of the voter $i$ and $H_i$ for its history. Say that you also have an "expertise" parameter $\mu_i$ such that $\Pr(A=V_i) = \mu_i$. If you put some prior on these $\mu_i$ -for example a Beta prior- you should be able to use the Bayes theorem to infer $\Pr(\mu_i \mid H_i)$, and then integrate over $\mu_i$ to compute $$\Pr(A \mid V_i, H_i) = \int_{\mu_i} \Pr(A, \mu_i \mid A_i, H_i)~ \mathrm{d}\mu_i$$

These systems are difficult to solve. You can use the EM algorithm as an approximation, or use complete likelihood maximisation scheme to perform exact Bayesian inference.

Take a look on this paper Variational Inference for Crowdsourcing, Liu, Peng and Ihler 2012 (presented yesterday at NIPS !) for detailed algorithms for solving this task.

share|improve this answer
Thanks for your answer, but could you please be a bit more specific about it? In particular, what you mean by expertise? If it's just probability that the person will answer correctly, then we already have its estimate as an average of previous answers, so it's not latent. If you mean than expertise incorporates both average and confidence about our estimate, then how can we propagate probabilities to get expertise and result? – ffriend Dec 6 '12 at 8:21
Yes, you can represent both average and confidence with this "expertise" variable and Bayesian inference. I have added a few explanations and a reference to my answer. Hope that helps ! – Emile Dec 7 '12 at 1:04
Great! This is exactly what I need. Thank you! – ffriend Dec 10 '12 at 8:46

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.