In a population of $N$, $K$ experts pick $M_{k\in\{1, ..., K\}}$ individuals that will have a certain attribute. Note that $M$ can be different across experts (e.g., one expert can pick 5 individuals, another can pick 20). My goal is to sort the individuals by their relative likelihood of having that attribute. First thought that came to mind was to use Bayesian inference, but I don't have a history of how well the experts have done in the past. How would you formulate this problem?

[Edit] A bit about the data set --- a close analogy to my problem is in fantasy football. At the start of every season, a panel of "pundits" pick their best XI to watch. My job is to aggregate all the picks without knowing each expert's track record. All I have is a bunch of IDs for each expert's picks but I can theoretically work out how the experts arrive at their selection by regressing on players' height, weight, shirt number (if it matters), performance in previous season, club, country etc. I guess one restriction is that I need to combine experts' picks, rather than regressing past performance on player attributes and build my own model.

  • $\begingroup$ How exactly does your data look like? For each expert, you just have a bunch of IDs of the picked individuals and that's all, or do you have some additional data (e.g. the attributes used by experts)? $\endgroup$
    – Tim
    Jul 19, 2021 at 7:39
  • $\begingroup$ Hi Tim, added a bit more context on the data set. I only have a bunch of IDs but I do have access to external attributes that the experts may or may not have used. $\endgroup$
    – stevew
    Jul 19, 2021 at 8:02
  • $\begingroup$ What you mean by "without knowing each expert's track record"? $\endgroup$
    – Tim
    Jul 19, 2021 at 8:03
  • $\begingroup$ As in I don't know accurate each expert is in the past. $\endgroup$
    – stevew
    Jul 19, 2021 at 8:04
  • $\begingroup$ Why does it matter? $\endgroup$
    – Tim
    Jul 19, 2021 at 8:15

1 Answer 1


I understand that you worry that you don't know the historical performance of the experts, so you don't know how reliable are their responses. It's a valid worry, but think of it in terms of wisdom of a crowd, their aggregate opinion should outweigh their individual biases and mistakes. For this to work, you would want the opinions to be made independently of each other (experts don't consult each other opinions), and best if the experts would not be a very heterogeneous group, though those are rather nice-to-haves. So you can just count the votes and sort the individuals by counts.

But let's say that you worry that some of the experts can be biased, for example, they tend to be very generous when voting, or the opposite, they pick very few individuals by default. You might want to use a statistical model to correct those biases. One of the models used in such cases is the Rasch model

$$ P(Y_{ij} = 1) = \frac{\exp(\theta_i - \beta_j)}{1+\exp(\theta_i - \beta_j)} $$

Where the probability of seeing the $i$-th individual picked by $j$-th expert is coded as a binary variable $Y_{ij}$ and modeled in terms of the two parameters for the true attribute $\theta_i$ of the individual and the expert's bias $\beta_j$. Notice that the model is just a generalized mixed-effects logistic regression model. You can use such a model for your data and treat the $\theta_i$ as the "distilled" attribute scores. There are also more complicated Item Response Theory models, such models are commonly used for modeling what is called the examiner effect, I encourage you to search for the term to learn more.

Notice that this is very similar to matrix factorization, used by recommender engines. In matrix factorization, you would decompose the $n \times k$ matrix of the binary picks by $k$ experts for $n$ individuals into two components $P$ and $Q$ of the dimensions $n \times m$ and $m \times k$

$$ Y \approx PQ $$

In such a case, you created embedding vectors for the data, where $P_{i\cdot}$ can be thought of as a vector of $m$ latent features describing the individual, relevant to the $Y_{i\cdot}$ votes they got. In the recommender system, you could use the embeddings for making the "people similar to you liked this" kind of recommendations. Here you don't need such a feature, as you don't really need to "recommend" the experts, but still, the $Y_{i\cdot}$ vector can be used for ranking or clustering the individuals. I'm describing this to show how a similar idea is used in a different field, so it can serve as an inspiration. The recommender systems literature can also give you some hints about scaling this if you are dealing with a big sample size $n$.

  • $\begingroup$ thanks very much for your detailed answer. So I've looked up the Rasch model (en.wikipedia.org/wiki/Rasch_model) and the version in Wiki has denominator $1 + exp(\theta_i - \beta_i)$. Why is there a difference? Also, how would you go about estimating the parameters of the model (after a bit of googling, looks like it's MLE)? $\endgroup$
    – stevew
    Jul 20, 2021 at 0:37
  • $\begingroup$ @stevew I did a typo, should be +. For estimation, check ppw.kuleuven.be/okp/_pdf/DeBoeck2011TEOIR.pdf , but you could treat it as a fixed-effects logistic regression and just use MLE. $\endgroup$
    – Tim
    Jul 20, 2021 at 5:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.