How should I test a function which gives a measure of how well objects fit a category? I am creating a function that gives a measure of how well a given object fits a specific category. The function is intended to give a measure that predicts how well in general humans think the object fits the category. The question I have is, assuming that I've created such a function, how do I test its validity?
I am intending to collect human judgements on a collection of pairs of objects where the subjects will label which object from the pair better fits the category. I want to show that the function accurately predicts when humans will find one object a better category member than another. I think what is confusing me is that the humans won't always agree so the problem is a matter of degree and I'm not sure how to account for that.
A simple test I am considering is to test for each pair whether the function gives a higher score to the object which most humans think best fits the category. I can then calculate the agreement of this (using Cohen's kappa?). This however is a bit crude and doesn't account for the fact that in situations where humans agree strongly (e.g. 95% pick object A and 5% pick object B) the function should find a much higher score for A than for B, also where humans aren't strongly in agreement e.g. 55% pick A and 45% pick B, the function should give similar scores to A and to B.
Can anyone enlighten me on how to go about testing this more fully? I realise that this might be quite a basic question so if anyone has suggested reading rather than answers that would also be great :)
 A: Your human subjects will yield probabilities between zero and one, namely the percentage of people who believe object A best fits the category. So your function should also output such a percentage or probability.
If your function is "good", then it should be able to predict this percentage correctly. You can assess this predictive capability either on aggregate data, or on granular data.


*

*If you have aggregate data (many people evaluating each pair, and you are only interested in predicting the aggregate outcome), then you can use standard prediction accuracy measures, like the mean squared error between your probabilistic predictions and the actual percentages.

*If you have granular data, things are a bit more interesting. In this case, you predict the probability that any given individual will pick object A. (For instance, your prediction might not only depend on the objects and the categories, but also on characteristics of the test subjects.) In this case, you have to evaluate probabilistic predictions for binary events. The best way to do this is to use proper scoring-rules. The tag wiki contains a number of pointers to literature. For the binary case specifically, I recommend a manuscript by Buja et al. (2005, "Loss Functions for Binary Class Probability Estimation and Classification: Structure and Applications").

