1
$\begingroup$

I have a document corpus which I would like to query to find documents which have the same or similar concepts. This is different from a keyword search in that the term would not need to appear in the document itself, but could possibly be returned because it has terms which tend to co-occur or be related to the search term.

Are there any open-source concept extraction and searching algorithms which I could adapt? What is the most popular?

$\endgroup$
1
$\begingroup$

The most straightforward approach to your problem consists of finding document features and then finding the most similar document to features extracted from queries (using kNN for example). You can obtain these features using:

Matrix decomposition

There is a simple method that does exactly what you described called Latent Semantic Analysis. It factorizes term-document matrix using SVD, so it effectively finds directions in the space defined by words/terms which correspond to groups of words, which usually correspond to concepts. Another similar method is Nonnegative Matrix Factorization (it could be interpreted as soft clustering).

You can find these methods in Python in scikit-learn library. LSA can be performed with Truncated SVD and NMF is just NMF.

Word embeddings

Another approach would be to use neural network-based methods, but these methods are more arcane, and there are lots of them. Also many of them require complex pipelines to load data and are language-specific (note that decomposition using methods are not).

For examples that you can just simply run Universal Sentence Encoder comes to mind. This Tensorflow version seems to have very simple interface (you just put strings you want to encode, it also doesn't seem to require any Tensorflow skills).

Other

You can also use topic models, which in some cases work like matrix factorization (for example NMF can be used for topic modelling), but I'll skip these methods since user3554004 covered them.

$\endgroup$
0
$\begingroup$

This could be solved using either latent topic modelling (cf. Latent Dirichlet Allocation, or just TFIDF), if you are interested in rather finding similar documents.

If you are looking for ocurrences of phrases or words or sentences that are similar in meaning to some list of terms and/or phrases, then what you are looking for is a distributional aka vector semantics model. You could either download pre-computed embeddings (e.g. word2vec ones from google, fasttext from facebook, etc) if it's just words and the domain is fairly general. Or if the domain is something specific or you want more control over the model, then train a model yourself (given the python tag: look into gensim for both that and LDA). The search portion of the task is then is finding words in the target corpus that are similar (in the vector space) to your list of words, and picking the documents where they occur.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.