I have 630 data points on anonymous participants' personalities, which is a five-dimensional array: Openness, Consciousness, Extroversion, Agreeableness and Neuroticism (known as the "Big Five" test, a well-established inventory in social science.) For each of those entries, the participant also identified which of four animals he or she most identifies with: A lion, a hawk, a butterfly or a dolphin.
That may seem kind of silly, but when I run boxplots on the five variables aggregated by favorite animal, I see some clear correlations. Each animal has 2-3 personality traits with low standard deviations in the responses (which range from 1-7, but which I normalized to 0 to 1), as well as 2-3 that bear no apparent correspondence.
My goal is to allow anyone to take this personality test and tell them how well they identify with each animal. I originally mistook this as a classification problem, and used R's
caret library to create a fairly accurate model that correctly predicted most of the animals in the test set based on the training set.
But that doesn't solve the problem, because the goal is not to tell the user that she or he MOST identifies with one animal, but give him or her a value from 0 to 100 that represents how well he or she matches which each of the four. Some people will fall right between a lion and a hawk, for example.
I've tried a few things. One was to take the Euclidean distance from a test user's data and measure the distance to the median values for each animal. But this does not work because not all five values have the same weight for a given animal. A lion has consistently low Agreeableness and high Extroversion, but Openness is all over the map.
Next, I tried to create four probability-based classification models use a GBM model, one for each animal, and then look at the probability that a user matches each of the four. This worked a little better, but did not produce remotely equal values for test users who appear to have a high affinity for two animals.
My question is what the appropriate model is for measuring the closeness of user's five values to each of four different animals, bearing in mind that each animal has only 2-3 of the five personality values that highly correlate. I tried taking a weighted Euclidean distance, creating weights based on the correlation of each trait, but this had extremely low efficacy in correctly assigning the highest value--which is to say, the lowest distance--to more than about 25 percent of the test set's preferred species.