0
$\begingroup$

I'm trying to match strings of words with a website which has bulletpoints from all of the URL's I'm interested in whose text is most similar to it. The way I thought of doing it is to get all of the documents from each bulletpoint into one corpus per website that I want to match a string of words with, discarding stop words and lemmatizing everything. Then, for each string of text, I create a TF-IDF sparse matrix, with each row the text from a single bulletpoint from a single website, so that the matrix contains all the text from the bulletpoints from all the websites, as well as a row for the string of words I want to match.

How should I then decide which row my string of words is most similar to? Should I get the cosine similarity of every row with my string of words row and just take whatever one has the highest cosine similarity (I'll have a way of identifying the row with the website it was scrapped from)? Or is there an actual formalized way to go about this once I have my sparse matrix?

$\endgroup$

1 Answer 1

0
$\begingroup$

If your string of words is not weighted (no hierarchy of most important to least important word), tf-idf-weighting and desparsing is not really necessary. You are only interested in the words in your string, so all other words may be disregarded.

Just compose a document x relevant terms tf-matrix. Then, divide each column (the count for each word) by the maximal number of times this word occurred, thereby normalizing each word count to a 0 (did not occur) to 1 (occurred most frequently) score. By summing up the scores for each document, you get the highest score for the most relevant document.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.