# Voting system that uses confidence for each voter

Let's say, we have simple "yes/no" question that we want to know answer to. And there are N people "voting" for correct answer. Every voter has a history - list of 1's and 0's, showing whether they were right or wrong about this kind of questions in the past. If we assume history as a binomial distribution, we can find voters' mean performance on such questions, their variation, CI and any other kind of confidence metrics.

Basically, my question is: how to incorporate confidence information into voting system?

For example, if we consider only mean performance of each voter, then we can construct simple weighted voting system:

$$result = sign(\sum_{v \in voters}\mu_v \times (-1)^{1-vote})$$

That is, we can just sum voters' weights multiplied either by $+1$ (for "yes") or by $-1$ (for "no"). It makes sense: if voter 1 has average of correct answers equal to $.9$, and voter 2 has only $.8$, than, probably, 1st person's vote should be considered as more important. On other hand, if 1st person have answered only 10 questions of this kind, and 2nd person have answered 1000 such questions, we are much more confident about 2nd person's skill level than about those of the 1st - it's just possible that 1st person was lucky, and after 10 relatively successful answers he will continue with much worse results.

So, more precise question may sound like this: is there statistical metric that incorporates both - strength and confidence about some parameter?

-

Let's denote the variables $A$ for the true answer, $V_i$ for the vote of the voter $i$ and $H_i$ for its history. Say that you also have an "expertise" parameter $\mu_i$ such that $\Pr(A=V_i) = \mu_i$. If you put some prior on these $\mu_i$ -for example a Beta prior- you should be able to use the Bayes theorem to infer $\Pr(\mu_i \mid H_i)$, and then integrate over $\mu_i$ to compute $$\Pr(A \mid V_i, H_i) = \int_{\mu_i} \Pr(A, \mu_i \mid A_i, H_i)~ \mathrm{d}\mu_i$$