For simplicity let's say we have an alphabet of ABC and we are looking at words that all have the same length (n = 10). Then the dataset can look something like this:


My dataset will contain ~20% true signal (similar words), whereas the remaining 80% percent are just "random" letters. Notably I will assume that the letters are independent and their distances the same.

What I thought of

Something iterative
I thought of something iterative, like starting with a set of words and than adding words. And after each addition removing the once that attribute the least to the pattern. Perhaps similar to what was suggest here for protein sequences.

Word clustering
For example clustering with Levenshtein as mentioned here (although I think this will be way to computer intensive for > 20.000 character vectors)

Is there an algorithm/method that can be used to enrich signal from a huge dataset (~22,000)?

Hopefully it is somewhat clear what I'm trying to achieve. I have little knowledge in statistics hence I'm not exactly sure what to mention here, so feel free to comment

  • $\begingroup$ Perhaps you could try calculating the entropy of each string and then clustering the strings based on the entropy value? I dunno if this is done at all in text mining; it's just something you could try. See this recent question for some answers on how to calculate it (a search for information entropy would be good too) $\endgroup$ – InfProbSciX Oct 13 '18 at 17:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.