0
$\begingroup$

Information:

So I have a data set with 18 vectors with 167 components, each of with has a value with a range of $[-2, 2]$. I am trying to calculate the similarity between one arbitrary vector in this data set and another arbitrary vector in the data set, and so far I have tried using two methods: Spearman's Footrule Distance and Jaccard Similarity (by detecting if the values at position n in each vector was greater than zero; if yes, then it returns 1). Unfortunately, there are some outlier vectors in the data set (which must be included) in which a disproportionate amount of values are, say zero or less than zero, or larger than zero (in which case this outlier will be #1 in similarity in almost every case). In this situation, the value returned for the outliers by the methods I am using is far smaller than the values returned for the non-outliers, thus making the outliers the most dissimilar vector for each different vector tested.

What I need:

Is there either a specific type of similarity I can use or is there a way to adjust/normalize the data set so that this doesn't happen?

Context for the data:

Each vector is for a specific person, and the components are ratings of songs.

Link to data: https://docs.google.com/spreadsheets/d/12CXk-vzJxYaEhD1QsAXx25JRDDubnqV9zAvG2sc1ykw/edit#gid=1484196880

EDIT: Added comment: [The goal is] to see which people were similer. There are outliers that rated a large amount of songs as a -2 or a 2, for example, and are thus either very similar to all of them or dissimilar

$\endgroup$
2
  • $\begingroup$ I don't understand. 18 people rated 167 songs, right? What is your aim? To see which people were similar? Which songs? Or what? And how are there outliers when the variable is -2 to 2 (sounds like a Likert type scale). $\endgroup$
    – Peter Flom
    Aug 16 '19 at 11:36
  • $\begingroup$ @PeterFlom To see which people were similer. There are outliers that rated a large amount of songs as a -2 or a 2, for example, and are thus either very similar to all of them or dissimilar. $\endgroup$
    – Jodast
    Aug 16 '19 at 22:21
1
$\begingroup$

It's going to depend on what you mean by "similar". You could, for instance, make every person have a mean of 0 for the songs by subtracting the mean for each person from every rating for that person. That works well if what you mean by "similar" is how the songs are rated, relative to each other.

But is that what you want? A person who rates nearly every song badly is, in a sense, most similar to other people who do that.

You can calculate the distance between vectors in various ways - absolute value and squared value are two common ones, but there is no reason you could not use some other function, as long as you can justify it - and these will have different results. (There are some requirements for calling it a "distance", so you might want some other term, if you violate those requirements).

Is someone who gives every song a -2 more similar to

a) A person who gives every song a 0 b) A person who gives most songs a -2 and some songs a different value c) A person who gives every song a -1.5?

and so on.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.