Given a sorted set of data points, where each point represents the length of a common substring between two files, I want to systematically decide a cut off point where I can start considering them to be "short" and no longer care about them.
Basically my goal is to exclude the data points that represent commonalities between the two files that exist because we’re looking at such small substrings the probability that they will repeat is large, rather than because they are actual common substrings.
My attempt to solve this was to first make a simple histogram using the Freedman-Diaconis rule for choosing a bin width. An example here shows the bucket number on the left and the frequency on the right.
0 : 68
1 : 14
2 : 5
3 : 6
4 : 1
5 : 0
6 : 1
7 : 1
8 : 1
9 : 1
10 : 2
11 : 2
You can see from above that the first couple buckets are littered with these small commonalities I am wishing to avoid. I don't want to have a "hard coded" cut off point but would like to decide this for each data set. Any ideas or pointers based on what I mentioned as far to solve this? Also I would like some input into my idea of using a histogram to solve this. If any other methods are better/more practical or any improvements to what I wrote I would love to hear.
