2
$\begingroup$

I have a set of documents from a wide variety of topics and I would like to retrieve the ones that are more similar to a new document provided. A search based on common words is not good enough, so traditional methods like TF-IDF are not what I am looking for. Moreover, I am looking for an unsupervised way to do the search.

So far, I've tested with Word Mover's Distance and similarity measures based on sentence embeddings like Smooth Inverse Frequency, but the quality of the results was not sufficient.

I've also done some research on the newest trendings in NLP for vector-space representations of text: BERT and MT-DNN. These solutions are the state of the art for text similarity only as a task that needs to be trained in a supervised manner downstream. I could not find any successful use of these embeddings to calculate a similarity measure directly.

Finally, I have found a solution that is very satisfactory for my requirements, it is available online at cortical.io. However, it is a closed solution and it is not available for the language that I need (Portuguese). Is there any open-source solution similar to this one that I am missing?

$\endgroup$
  • $\begingroup$ Is your goal to determine common authorship? If so, the following article may be of interest to you: journal.r-project.org/archive/2016-1/eder-rybicki-kestemont.pdf $\endgroup$ – StatsStudent Feb 13 at 21:15
  • $\begingroup$ No, my goal is to calculate the similarity based on the text content, not who is the author. $\endgroup$ – Pedro Igor A. Oliveira Feb 14 at 12:20
  • $\begingroup$ OK. But to be clear, stylo examines document text and tries to group the documents into the same author based on the content, but in reality, it could be used if the author was unknown -- you could train it with, say document A, and then say, look at the other documents and find ones that are stylistically similar to document A. $\endgroup$ – StatsStudent Feb 14 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.