How should I test a function which gives a measure of how well objects fit a category?

I am creating a function that gives a measure of how well a given object fits a specific category. The function is intended to give a measure that predicts how well in general humans think the object fits the category. The question I have is, assuming that I've created such a function, how do I test its validity?

I am intending to collect human judgements on a collection of pairs of objects where the subjects will label which object from the pair better fits the category. I want to show that the function accurately predicts when humans will find one object a better category member than another. I think what is confusing me is that the humans won't always agree so the problem is a matter of degree and I'm not sure how to account for that.

A simple test I am considering is to test for each pair whether the function gives a higher score to the object which most humans think best fits the category. I can then calculate the agreement of this (using Cohen's kappa?). This however is a bit crude and doesn't account for the fact that in situations where humans agree strongly (e.g. 95% pick object A and 5% pick object B) the function should find a much higher score for A than for B, also where humans aren't strongly in agreement e.g. 55% pick A and 45% pick B, the function should give similar scores to A and to B.

Can anyone enlighten me on how to go about testing this more fully? I realise that this might be quite a basic question so if anyone has suggested reading rather than answers that would also be great :)

• Say $\alpha$ is the score for A and $\beta$ is the score for B, we expect that if $\alpha = \beta$ 50% would pick A and 50% B, also if $\alpha = 1$, $\beta = 0$ then 100% would pick A and 0% B (and vice versa). Also we expect the score for A to be as important as the score for B. From my understanding, a linear regression model would give something like $0.5 \alpha - 0.5\beta + 0.5$. I see this is oversimplified as there probably isn't a linear relationship between the difference in $\alpha$ and $\beta$ and the percentage of subjects picking A over B, but is this not a reasonable assumption? May 9 '19 at 15:30
• Since you are modeling a binary response, don't use "ordinary" regression, but logistic regression. Probably include $\alpha-\beta$ as a predictor. Which would still treat the response as linear in the logit link, so splines may (may!) still be useful. May 9 '19 at 15:54
• Ok, so in this case I would take a sample of my data and perform logistic regression to calculate a function of $\alpha - \beta$ which predicts the percentage of humans picking A or B. I could then use this function to test my original function on the rest of the data set in the way you initially described? Also, I thought that if I consider the responses as an aggregate then I am not using a binary response and could therefore use "ordinary" regression? May 9 '19 at 16:12