2
$\begingroup$

I have records (rows) in a database and I want to compute similar records. I have a constraint to use cosine similarity. If the variables (attributes, columns) vary in type and come in this form:

[number] [number] [boolean] [20 chars string]

how can I proceed to the vectorization to apply the cosine similarity? For the string I can take the simple tf-idf. But for numbers and boolean values?. And how can this be combined?

|cite|improve this question
$\endgroup$
  • $\begingroup$ Cosine for boolean (binary) data is called sometimes Ochiai coefficient and has its formula, but general cosine formula of course is valid too. So, the only ambiguity remaining is with that string variable. tf-idf (as far as I know) isn't cosine. Well, what are similar and what are dissimilar records by that string for you?. Describe it - perhaps with examples. $\endgroup$ – ttnphns Mar 19 '13 at 17:34
  • $\begingroup$ I mean how to proceed to vectorization. Of course tf-idf isn't cosine. It's a way to vectorize a text. Can you be precise how to vectorize boolean and numbers in order to construct the vectors and fed the cosine with them $\endgroup$ – curious Mar 19 '13 at 17:37
  • $\begingroup$ What do you mean under vectorization? If it's unfolding a matrix into a vector then I don't see why you need it. If you have any data matrix records X numeric_attributes then you will be able to obtain a square symmetric matrix of cosine similarity between the records. $\endgroup$ – ttnphns Mar 19 '13 at 17:44
  • $\begingroup$ Vectorization is the first step of cosine similarity.Suppose i have two records. r1=234,1023,No,Today is Sunday. and r2=876,423,Yes,Tomorrow i am leaving. How i can compute the cosine of those 2 records?How i can compute their vectors?I will just take char by char their ascii representation and make a vector? Then there is no semantic and cosine might give inaccurate results $\endgroup$ – curious Mar 19 '13 at 17:52
  • $\begingroup$ If in your comment example one omits the last, string attribute (because cosine cannot be computed unless all the data are numeric) then the raw cosine (cosine computed on not anyhow standardized data) between r1 and r2 is .62468. $\endgroup$ – ttnphns Mar 19 '13 at 18:06
0
$\begingroup$

You can normalize each field - divide by mean value, etc. You can also weight normalized fields based on their importance from domain knowledge. after this you can apply standard cosine similarity math.

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.