What are the common ways to define proximity between two sentences ? I did various tries, including:

either using the counts for the Jaccard and Cosine "metrics" or the TFIDF for the cosine similarity. They performed relatively well.

Some (string specific) kernels can be adapted as well:

And general kernels can be applied (Polynomial, Gaussian, Information diffusion...)

Am I missing any possible similarity measure ? Is any state-of-art in defining metrics over texts available ?

  • 1
    $\begingroup$ "Diffusion Kernels on Statistical Manifolds" by Lafferty and Lebanon develops a kernel function which the authors claim is more powerful than linear and RBF kernels. Note that all the methods you've outlined are just string similarity or bag-of-words methods. So you could add some value by looking into part-of-speech tagging methods and n-grams. $\endgroup$ – Sycorax Oct 19 '15 at 16:47
  • $\begingroup$ @user777 Thank you! I forgot to mention that I tried ID-kernel it as well. It gave better results than a simple cosine, actually. Do you have more references regarding "part of speech tagging?" $\endgroup$ – RUser4512 Oct 19 '15 at 16:50
  • $\begingroup$ You might start here. en.wikipedia.org/wiki/Part-of-speech_tagging $\endgroup$ – Sycorax Oct 19 '15 at 16:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.