Machine learning model for measuring distance of five variables to four possible outputs

Question

I have 630 data points on anonymous participants' personalities, which is a five-dimensional array: Openness, Consciousness, Extroversion, Agreeableness and Neuroticism (known as the "Big Five" test, a well-established inventory in social science.) For each of those entries, the participant also identified which of four animals he or she most identifies with: A lion, a hawk, a butterfly or a dolphin.

That may seem kind of silly, but when I run boxplots on the five variables aggregated by favorite animal, I see some clear correlations. Each animal has 2-3 personality traits with low standard deviations in the responses (which range from 1-7, but which I normalized to 0 to 1), as well as 2-3 that bear no apparent correspondence.

My goal is to allow anyone to take this personality test and tell them how well they identify with each animal. I originally mistook this as a classification problem, and used R's caret library to create a fairly accurate model that correctly predicted most of the animals in the test set based on the training set.

But that doesn't solve the problem, because the goal is not to tell the user that she or he MOST identifies with one animal, but give him or her a value from 0 to 100 that represents how well he or she matches which each of the four. Some people will fall right between a lion and a hawk, for example.

I've tried a few things. One was to take the Euclidean distance from a test user's data and measure the distance to the median values for each animal. But this does not work because not all five values have the same weight for a given animal. A lion has consistently low Agreeableness and high Extroversion, but Openness is all over the map.

Next, I tried to create four probability-based classification models use a GBM model, one for each animal, and then look at the probability that a user matches each of the four. This worked a little better, but did not produce remotely equal values for test users who appear to have a high affinity for two animals.

My question is what the appropriate model is for measuring the closeness of user's five values to each of four different animals, bearing in mind that each animal has only 2-3 of the five personality values that highly correlate. I tried taking a weighted Euclidean distance, creating weights based on the correlation of each trait, but this had extremely low efficacy in correctly assigning the highest value--which is to say, the lowest distance--to more than about 25 percent of the test set's preferred species.

Thanks much!

Answer 1

I think you were on the right track with your first approach. You just need to output the individual class probabilities instead of the predicted class, and they should all equal 100 when added up, so it scales nicely to what you want to do.

Your comment about Euclidean distances, and the fact that your input variables sound like they're all on the same scale, got me thinking.... A really cool model to try would be a nearest neighbor approach. Plot all 630 people in a 5 dimensional space. Then for each 1, find the 10 nearest neighbors using euclidean distance. Then count the animals for those 10 nearest neighbors. Multiply by 10 to have it on a scale of 100.

To test if it works, just use those 10 neighbors to vote on a winning animal. Check see if that classification accuracy is close to your original models accuracy.

At the end of the day, it might be easier to just research how to get your R model to output the individual class probabilities.