1
$\begingroup$

I have a file with a list of product descriptions. These product description are long strings. Eg "These blue pants are a resistant and comfortable product for tracking and ciclying. In the picture we can see the long model"

I have another file (much bigger) with product name. Eg "long sport blue pants"

I have thought about counting the number of words in the titles that contain words from the description and matching the description with the title that contains the most words. Does it make sense? Is there an algorithm for that in R? Is there any other way to do this kind of matching?

Thank you!

$\endgroup$
3
$\begingroup$

I would try to use cosine similarity between two descriptions using the Vector Space Models, and match ones with highest similarity

In R the transformation to the vector space would look like this:

library(tm)

shortDesc = all$Headline
    longDesc = all$Abstract

corpus = Corpus(VectorSource(c(shortDesc, longDesc)))
corpus = tm_map(corpus, tolower)
corpus = tm_map(corpus, PlainTextDocument)
corpus = tm_map(corpus, removePunctuation)
corpus = tm_map(corpus, removeWords, stopwords("english"))

dtm = DocumentTermMatrix(corpus)
m1 = dtm[1:100, ]
m2 = dtm[101:150, ]

Here m1 is the document matrix for the short description and m2 is the matrix with long descriptions

And now you can use cosine for matching. I don't seem to be able to find an implementation of the cosine similarity function for the tm package, but luckily it's very easy to implement this function ourselves

library(slam)

cosine = function(m1, m2=m1) {
  norm_m1 = row_sums(m1 ^ 2)
  norm_m2 = row_sums(m2 ^ 2)

  cross = as.simple_triplet_matrix(sqrt(outer(norm_m1, norm_m2)))
  tcrossprod_simple_triplet_matrix(m1, m2) / cross
}

The tm library relies on the slam package for the sparse matrices implementation, so we can use it as well and avoid converting to the dense representation. It probably should be faster this way.

Finally, let's use the cosine function for matching:

simMatrix = cosine(m1, m2)
matching = apply(simMatrix, MARGIN=1, FUN=which.max)
df = data.frame(short=shortDesc, long=longDesc[matching])

Now the dataframe df contains the short description and the best matching long description

$\endgroup$
  • $\begingroup$ Hi Alexey, thank you for your answer. This line Corpus(VectorSource(shortDesc), VectorSource(longDest)) returns a corpus whose length is the same as my shortDesc. Does it ignore long Dest all together? or does it extract the terms from longDest? $\endgroup$ – DroppingOff May 6 '15 at 19:01
  • $\begingroup$ and another question. cos.doc = cosine(as.matrix(corpus.dtm)) returns a matrix of the number of terms in corpus x number of terms in corpus. How do I use that extract the distance or similarity? $\endgroup$ – DroppingOff May 6 '15 at 19:03
  • $\begingroup$ I updated the answer, I hope it answers the questions $\endgroup$ – Alexey Grigorev May 6 '15 at 20:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.