3
$\begingroup$

I have been using Latent Semantic Analysis (LSA) or Latent Semantic Indexing (LSI) to identify whether different email addresses belong to the same individual by matching on names used for each email address; An email address represents a 'document', and the set of all names (as separate words) are the terms in that document:

"John Doe" < johnd@example.com > is converted into the document 'johnd@example.com' with terms ['john', 'doe', 'johnd'].

The typical LSA/LSI application more or less applies these techniques:

  • Construct term-document matrix
  • Apply tf--idf model
  • Compute SVD; Perform rank/dimension reduction.
  • Compute cosine similarity

There have been multiple studies showing the above technique is quite successful in matching documents. However, my data does not exist of actual (English) document, but only a few words; Namely the names of the people that have sent emails.

To give you an idea, my data set consists of 99,012 terms and 76,580 documents. The average number of terms in a document is 2.70.

To create a relation in the LSA/LSI application between two documents that are mutually exclusive but should match (e.g. as a result of misspelling), I have decided to augment the term-document matrix with an edit distance such as the Levenshtein distance.

I have searched the web and have not been able to find any website that describes the use of LSA/LSI in combination with edit distance. Is there a reason this has not been applied or described before? Does anybody have more information on combining LSI/LSA and edit distance? I can hardly believe I am the first that has applied the combination of these techniques.

I did find two duplicate questions which never received an answer:

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.