For simplicity let's say we have an alphabet of
ABC and we are looking at words that all have the same length (
n = 10). Then the dataset can look something like this:
[,1] A "ABCCACCAAB" B "ABBBCAAABA" C "CCCABBAABC" D "AABBCCBACA" E "BCACCBBCAA"
My dataset will contain
~20% true signal (similar words), whereas the remaining
80% percent are just "random" letters. Notably I will assume that the letters are independent and their distances the same.
What I thought of
I thought of something iterative, like starting with a set of words and than adding words. And after each addition removing the once that attribute the least to the pattern. Perhaps similar to what was suggest here for protein sequences.
For example clustering with Levenshtein as mentioned here (although I think this will be way to computer intensive for > 20.000 character vectors)
Is there an algorithm/method that can be used to enrich signal from a huge dataset (~
Hopefully it is somewhat clear what I'm trying to achieve. I have little knowledge in statistics hence I'm not exactly sure what to mention here, so feel free to comment