Aggregating votes from different users with some constraints and probabilities Suppose we have $N$ users that can vote for a given question by "Yes" or "No". Each user have a probability which is considered as the confidence that we have in that user (e.g. a user may have a probability of 0.8 to answer the question correctly, and a probability of 0.2 to answer it wrongly). Each user requesst an amount of money as a reward for answering the question. I (as the asker of the question) have a limited budget for paying the users.
Given this budget, which users should I ask to maximize the final answer to be correct (i.e. the answer obtained after aggregating the asked answers)?
Additional informations:
"what does it mean for a vote to be correct": if the question is "does 10/2 equals 5?" then "Yes" is a correct answer to this question; now if the probability associated to the user is 1, that means that the user will answer this question correctly because it have a very good knowledge of the domain of the question (mathematics); if the probability associated to the user is 0, this means that the user will answer this question wrongly; if the probability associated to the user is 0.5, then his answer may be correct ("Yes 10/2=5") with a probability of 0.5, or maybe wrong wt probability 1-0.5.
What I want is to maximize the fact of getting the good final answer by aggregating the obtained answers (want to know that the answer to my question is "Yes" 10/2=5). For this, which users should I ask (given my budget, and given that each user request a given amount of money to accept to answer my question), and how to aggregate their answers in order to have the correct one.
What is called "vote" is the classical aggregation of the answers, by taking the most common answer among the received ones (if there is more "Yes" answers then ok I will consider the final result as "Yes), but in my real example I have some constraints: probabilities associated to users, ammount of money requested by each user to answer my question, and the total amount of money that I have, so I need to also consider these constraints. If you want, forget about the word "vote" in the question, just tell me who to ask and how to aggregate the answers that I get. 
 A: You can think of the "expected number of correct answers" for each user--for example, your hypothetical user with probability 0.8 of answering correctly has an expected number of correct answers equal to 0.8, when you give him a single question.
Then, divide by your cost for each user. Suppose your user with probability 0.8 asks for \$2 dollars. When you divide, you get an "expected number of correct answers per dollar" of 0.4 for that user. A user with probability 0.6 of answering correctly but who asked for only \$1 would have an "expected number of correct answers per dollar" of 0.6.
Now you have a few possible approaches. The simplest is to use a greedy algorithm, and go through an select as many users as you can within your budget starting with the highest "expected number of correct answers per dollar." This is an approximate algorithm.
A better approach is to treat this as a variant of the knapsack problem. Solving it is NP-hard, but there are polynomial time approximations which are better than the greedy algorithm above.
