I am creating a function that gives a measure of how well a given object fits a specific category. The function is intended to give a measure that predicts how well in general humans think the object fits the category. The question I have is, assuming that I've created such a function, how do I test its validity?
I am intending to collect human judgements on a collection of pairs of objects where the subjects will label which object from the pair better fits the category. I want to show that the function accurately predicts when humans will find one object a better category member than another. I think what is confusing me is that the humans won't always agree so the problem is a matter of degree and I'm not sure how to account for that.
A simple test I am considering is to test for each pair whether the function gives a higher score to the object which most humans think best fits the category. I can then calculate the agreement of this (using Cohen's kappa?). This however is a bit crude and doesn't account for the fact that in situations where humans agree strongly (e.g. 95% pick object A and 5% pick object B) the function should find a much higher score for A than for B, also where humans aren't strongly in agreement e.g. 55% pick A and 45% pick B, the function should give similar scores to A and to B.
Can anyone enlighten me on how to go about testing this more fully? I realise that this might be quite a basic question so if anyone has suggested reading rather than answers that would also be great :)