Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

First of all, my apologies if I mess up the terminology. I've been out of math for several years, so I'm certain I'm going to use terms incorrectly. Also, though I concentrated mathematics in college, I have lost those legs; I'm particularly braindead when it comes to remembering what means what with mathematical notations. If it wouldn't be too much to ask, sample calculations would be greatly appreciated where possible.

My situation:

I have multiple unbounded arrays which I would like to compare, where each element represents the frequency of the element (in the example below, colors):

Possible Elements are Colors: red, white, blue, orange, yellow, green

Set1: { red 20, blue 35, white 16 }
Set2: { red 15, white, 25, green 18 }
Set3: { white 12, yellow 3 }

I have so far tried measuring similarity by converting the arrays into n-dimensional vectors, normalizing them to unit vectors, and then calculating the cosine similarity. This method seems to work well, but there are certain situations where the measure seems to break, usually when one set is a lot smaller than another.

Set4: { red 20, blue 30 }
Set5: { red 1 }

Yields a 55.47% similarity between Set4 and Set5. 

Seeking to rectify this, instead of using the frequencies and normalizing them to unit vectors, I tried weighting the frequencies via TF-IDF. Additionally, the actual data distribution would conform to the power law, which made the logarithmic TF-IDF appealing. I was probably doing the TF-IDF calculation wrong, but the problem I ran into was that my document corpus was too small.

w = tf(t, d) * idf(t)

tf(t, d) is the term frequency in the document

and idf(t) = log ( Corpus Size / # of documents where term appears )
(denominator is adjusted with +1 if # of documents is 0)

Well, because I'm comparing the documents in pairs, I set my corpus size to 2. Then, frequently the # of documents is also "2" because elements frequently appear in both sets. This leads to idf(t) being log(2/2) = 0.

Which, technically speaking, means TF-IDF is working because it is supposed to penalize terms that appear in all documents. But this isn't the outcome I want.

So to sum up:

  • I have several arrays whose similarity I would like to compare
  • The frequency distribution of the elements in these arrays approached an exponential one (in case this matters).
  • I have tried taking the cosine similarity with two weighting methods: unit vectors calculated from the frequencies, and with TF-IDF.
  • Unit vectors worked kind of well, but broke with certain sets.
  • TF-IDF did not produce the results I wanted at all, though I was probably doing the calcs wrong.

My questions:

1) Was I plugging in the "corpus" and "count in documents" numbers correctly with TF-IDF? If not, what would be the proper figures to use for those variables because I don't have a fixed corpus size.

2) If I was plugging the variables in correctly, then is there a way to tweak TF-IDF so that it doesn't automatically nullify comparisons when the terms are shared by both documents? (Or in this case, when the elements are shared by both arrays?)

3) Is there a better similarity measure than cosine for what I am trying to do? Would it be possible, for instance, to use Dice's Coefficient in a manner similar to cosim? One concern I have, for instance, is whether I can pre-apply normalization/weighting if I want to use Dice.

4) Similarly, is there a better method I could use to weight the frequencies? Instead of using unit vectors or TF-IDF I mean.

share|improve this question

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.