I am creating a function that gives a measure of how well a given object fits a specific category. The function is intended to give a measure that predicts how well in general humans think the object fits the category. The question I have is, assuming that I've created such a function, how do I test its validity?

I am intending to collect human judgements on a collection of pairs of objects where the subjects will label which object from the pair better fits the category. I want to show that the function accurately predicts when humans will find one object a better category member than another. I think what is confusing me is that the humans won't always agree so the problem is a matter of degree and I'm not sure how to account for that.

A simple test I am considering is to test for each pair whether the function gives a higher score to the object which most humans think best fits the category. I can then calculate the agreement of this (using Cohen's kappa?). This however is a bit crude and doesn't account for the fact that in situations where humans agree strongly (e.g. 95% pick object A and 5% pick object B) the function should find a much higher score for A than for B, also where humans aren't strongly in agreement e.g. 55% pick A and 45% pick B, the function should give similar scores to A and to B.

Can anyone enlighten me on how to go about testing this more fully? I realise that this might be quite a basic question so if anyone has suggested reading rather than answers that would also be great :)


Your human subjects will yield probabilities between zero and one, namely the percentage of people who believe object A best fits the category. So your function should also output such a percentage or probability.

If your function is "good", then it should be able to predict this percentage correctly. You can assess this predictive capability either on aggregate data, or on granular data.

  • If you have aggregate data (many people evaluating each pair, and you are only interested in predicting the aggregate outcome), then you can use standard prediction accuracy measures, like the mean squared error between your probabilistic predictions and the actual percentages.
  • If you have granular data, things are a bit more interesting. In this case, you predict the probability that any given individual will pick object A. (For instance, your prediction might not only depend on the objects and the categories, but also on characteristics of the test subjects.) In this case, you have to evaluate probabilistic predictions for binary events. The best way to do this is to use proper . The tag wiki contains a number of pointers to literature. For the binary case specifically, I recommend a manuscript by Buja et al. (2005, "Loss Functions for Binary Class Probability Estimation and Classification: Structure and Applications").
  • $\begingroup$ Thanks for this. I guess the issue I have then is how to calculate this percentage from the function that I have. Currently the function just gives a score of how well an individual object fits the category (and this is the value that is important for later on), how would I go about using theses score to predict the percentage of people picking one object over another? $\endgroup$
    – A. Bollans
    May 9 '19 at 13:42
  • $\begingroup$ You could use the score in a logistic regression. Possibly consider a spline transformation, depending on how the score is calculated and how it hangs together with the actual propensity you are modeling. $\endgroup$ May 9 '19 at 13:51
  • $\begingroup$ Say $\alpha$ is the score for A and $\beta$ is the score for B, we expect that if $\alpha = \beta$ 50% would pick A and 50% B, also if $\alpha = 1$, $\beta = 0$ then 100% would pick A and 0% B (and vice versa). Also we expect the score for A to be as important as the score for B. From my understanding, a linear regression model would give something like $0.5 \alpha - 0.5\beta + 0.5$. I see this is oversimplified as there probably isn't a linear relationship between the difference in $\alpha$ and $\beta$ and the percentage of subjects picking A over B, but is this not a reasonable assumption? $\endgroup$
    – A. Bollans
    May 9 '19 at 15:30
  • $\begingroup$ Since you are modeling a binary response, don't use "ordinary" regression, but logistic regression. Probably include $\alpha-\beta$ as a predictor. Which would still treat the response as linear in the logit link, so splines may (may!) still be useful. $\endgroup$ May 9 '19 at 15:54
  • $\begingroup$ Ok, so in this case I would take a sample of my data and perform logistic regression to calculate a function of $\alpha - \beta$ which predicts the percentage of humans picking A or B. I could then use this function to test my original function on the rest of the data set in the way you initially described? Also, I thought that if I consider the responses as an aggregate then I am not using a binary response and could therefore use "ordinary" regression? $\endgroup$
    – A. Bollans
    May 9 '19 at 16:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.