I need to compare two datasets to see if they are "similar" in terms of numerical similarity between given columns. The problem is, however, that I do not know the characteristics of the 2nd dataset. It is possible the second dataset 1)matches fully to the 1st dataset, 2)matches partially to the 1st dataset 3) does not match at all. By matching, I mean whether the entities intersect or not. For instance, if I need to compare two datasets which contain country names and their respective GDP values, I have no idea, which countries exists in the second dataset, what the intersecting entities are, and whether the GDP values are in the same order of magnitude (in millions, billions, etc. ) beforehand. This makes it impossible to apply Cosine similarity, since the two vectors need to have the same length. Moreover, Cosine similarity does not consider entity-to-entity similarity. Using Euclidian distance is also not an option, because outliers will affect the overall similarity score.

I need a more complex similarity measure which can take into account all these unknown characteristics inherent to the 2nd dataset. The similarity measure I am looking for, should be robust to outliers and should somehow capture general, overall similarity between two datasets.

  • $\begingroup$ What would you do with this number if you had it? Suppose I have two datasets, say the dates of birth of US Presidents and a list of systolic blood pressures of 200 people. Is that an invalid comparison or should it return zero similarity? I think if you can say what you would do with this number, then you might understand your own question better. Otherwise my reaction is that this is too open a question to allow any precise answers, but i may be missing something. $\endgroup$ – Nick Cox Apr 14 '16 at 11:18
  • $\begingroup$ Where do you begin? You might quantify variables in common, records in common, values similar or not, etc. and even that is predicated on the idea that your dataset is one big table, and many are not. $\endgroup$ – Nick Cox Apr 14 '16 at 11:19
  • $\begingroup$ @NickCox Basically, since the US presidents' names and the names of the people in the blood pressure dataset would not match, the similarity should be zero. So, the largest weight should be given to the similarity score of the values of the intersecting entities. However, If you had 2 datasets, both with the US presidents as entities, but birth year and salary as the values in the 1st and the 2nd datasets respectively, then also the similarity should be zero as you cannot compare birth dates to salary. But usually it is not a problem, since I do compare matching attributes - GDPS to GDPS $\endgroup$ – Ahmedov Apr 14 '16 at 11:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.