Assume that there are k expert measuring some quality x with a number between zero and one, $x_i \in [0,1], i=1,2,...,k$. I would like to first know the aggregated quality of x with respect to the experts' opinions, and then I also want to know the extent to which the experts agree (or disagree).

To do so, I used the following model where x is the latent quality (similar to Kruschke, Chapter 9, Equation 9.4):

$ x_i \sim beta(x(k-2)+1,x(k-2)+1)$

where x is the mode of the beta, and $k>0$ is a constant. I know that the bigger value for $k$ means the more agreement between experts. But it does not show to what extent they agree.

  • $\begingroup$ What exactly is your data? Is it just $k$ measurements, one for each expert? How do you define agreement between the experts? It sounds like a small variance of the measurements, but without any other information or data it is rather hard to define what "small" means in here. $\endgroup$ – Tim Nov 8 '18 at 8:59
  • $\begingroup$ Thanks for reply. Yes, there k measurement, one for each expert. Let's say if all experts assign the same value, then there is full agreement. $\endgroup$ – Majid Mohammadi Nov 8 '18 at 10:00
  • $\begingroup$ Fast-and-dirty solution is to look how often do the experts differ by more then some $\varepsilon$. $\endgroup$ – Tim Nov 8 '18 at 10:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.