For the development of a new chemical process, a handful experts should be asked a few questions (4 to 6), what values they expect for several experimental parameters, i.e. temperature ranges and how confident they are regarding their guesses on a scale from 1 to 10.
I would like to weight their responses corresponding to the confidence.
Some people tend to be very confident with average ratings of 8, other tend to be less confident with ratings of 3 to 4.
To avoid such things as the Dunning-Kruger effect, I would like to get a rating of them, what confidence level means what and normalize their responses.
I thought of giving them a questionnaire with questions, they know for pretty sure, and others, where they most likely have to guess and take the confidence level their as well. If people give high confidence scores for questions, they could not possibly know, I can assume, that an 8 is more something other would rate with a 5.
I now wonder, if such a questionnaire to get the rater’s tendency in confidence scores already exists, possibly already widely evaluated?
I am not sure, if Cross Validated is the right corner of Stack Exchange to post this question, if not I am very sorry; feel free to delete this question then.
