Let's say, we have simple "yes/no" question that we want to know answer to. And there are N people "voting" for correct answer. Every voter has a history - list of 1's and 0's, showing whether they were right or wrong about this kind of questions in the past. If we assume history as a binomial distribution, we can find voters' mean performance on such questions, their variation, CI and any other kind of confidence metrics.

Basically, my question is: how to incorporate confidence information into voting system?

For example, if we consider only mean performance of each voter, then we can construct simple weighted voting system:

$$result = sign(\sum_{v \in voters}\mu_v \times (-1)^{1-vote})$$

That is, we can just sum voters' weights multiplied either by $+1$ (for "yes") or by $-1$ (for "no"). It makes sense: if voter 1 has average of correct answers equal to $.9$, and voter 2 has only $.8$, than, probably, 1st person's vote should be considered as more important. On other hand, if 1st person have answered only 10 questions of this kind, and 2nd person have answered 1000 such questions, we are much more confident about 2nd person's skill level than about those of the 1st - it's just possible that 1st person was lucky, and after 10 relatively successful answers he will continue with much worse results.

So, more precise question may sound like this: is there statistical metric that incorporates both - strength and confidence about some parameter?


You should consider the expertise of a voter as a latent variable of your system. You may then be able to solve your problem with bayesian inference. A representation as graphical model could be like this :


Let's denote the variables $A$ for the true answer, $V_i$ for the vote of the voter $i$ and $H_i$ for its history. Say that you also have an "expertise" parameter $\mu_i$ such that $\Pr(A=V_i) = \mu_i$. If you put some prior on these $\mu_i$ -for example a Beta prior- you should be able to use the Bayes theorem to infer $\Pr(\mu_i \mid H_i)$, and then integrate over $\mu_i$ to compute $$\Pr(A \mid V_i, H_i) = \int_{\mu_i} \Pr(A, \mu_i \mid A_i, H_i)~ \mathrm{d}\mu_i$$

These systems are difficult to solve. You can use the EM algorithm as an approximation, or use complete likelihood maximisation scheme to perform exact Bayesian inference.

Take a look on this paper Variational Inference for Crowdsourcing, Liu, Peng and Ihler 2012 (presented yesterday at NIPS !) for detailed algorithms for solving this task.

  • 1
    $\begingroup$ Thanks for your answer, but could you please be a bit more specific about it? In particular, what you mean by expertise? If it's just probability that the person will answer correctly, then we already have its estimate as an average of previous answers, so it's not latent. If you mean than expertise incorporates both average and confidence about our estimate, then how can we propagate probabilities to get expertise and result? $\endgroup$ – ffriend Dec 6 '12 at 8:21
  • $\begingroup$ Yes, you can represent both average and confidence with this "expertise" variable and Bayesian inference. I have added a few explanations and a reference to my answer. Hope that helps ! $\endgroup$ – Emile Dec 7 '12 at 1:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.