Bayesian combination of expert opinion In a population of $N$, $K$ experts pick $M_{k\in\{1, ..., K\}}$ individuals that will have a certain attribute. Note that $M$ can be different across experts (e.g., one expert can pick 5 individuals, another can pick 20). My goal is to sort the individuals by their relative likelihood of having that attribute. First thought that came to mind was to use Bayesian inference, but I don't have a history of how well the experts have done in the past. How would you formulate this problem?
[Edit]
A bit about the data set --- a close analogy to my problem is in fantasy football. At the start of every season, a panel of "pundits" pick their best XI to watch. My job is to aggregate all the picks without knowing each expert's track record. All I have is a bunch of IDs for each expert's picks but I can theoretically work out how the experts arrive at their selection by regressing on players' height, weight, shirt number (if it matters), performance in previous season, club, country etc. I guess one restriction is that I need to combine experts' picks, rather than regressing past performance on player attributes and build my own model.
 A: I understand that you worry that you don't know the historical performance of the experts, so you don't know how reliable are their responses. It's a valid worry, but think of it in terms of wisdom of a crowd, their aggregate opinion should outweigh their individual biases and mistakes. For this to work, you would want the opinions to be made independently of each other (experts don't consult each other opinions), and best if the experts would not be a very heterogeneous group, though those are rather nice-to-haves. So you can just count the votes and sort the individuals by counts.
But let's say that you worry that some of the experts can be biased, for example, they tend to be very generous when voting, or the opposite, they pick very few individuals by default. You might want to use a statistical model to correct those biases. One of the models used in such cases is the Rasch model
$$ P(Y_{ij} = 1) = \frac{\exp(\theta_i - \beta_j)}{1+\exp(\theta_i - \beta_j)} $$
Where the probability of seeing the $i$-th individual picked by $j$-th expert is coded as a binary variable $Y_{ij}$ and modeled in terms of the two parameters for the true attribute $\theta_i$ of the individual and the expert's bias $\beta_j$. Notice that the model is just a generalized mixed-effects logistic regression model. You can use such a model for your data and treat the $\theta_i$ as the "distilled" attribute scores. There are also more complicated Item Response Theory models, such models are commonly used for modeling what is called the examiner effect, I encourage you to search for the term to learn more.
Notice that this is very similar to matrix factorization, used by recommender engines. In matrix factorization, you would decompose the $n \times k$ matrix of the binary picks by $k$ experts for $n$ individuals into two components $P$ and $Q$ of the dimensions $n \times m$ and $m \times k$
$$
Y \approx PQ
$$
In such a case, you created embedding vectors for the data, where $P_{i\cdot}$ can be thought of as a vector of $m$ latent features describing the individual, relevant to the $Y_{i\cdot}$ votes they got. In the recommender system, you could use the embeddings for making the "people similar to you liked this" kind of recommendations. Here you don't need such a feature, as you don't really need to "recommend" the experts, but still, the $Y_{i\cdot}$ vector can be used for ranking or clustering the individuals. I'm describing this to show how a similar idea is used in a different field, so it can serve as an inspiration. The recommender systems literature can also give you some hints about scaling this if you are dealing with a big sample size $n$.
