Mathematically finding a threshold for deciding on rare word I am implementing spell-correction facility as a preprocessing step of for a text classification project. For this reason I have to make a knowledge-base where I shall be putting all the words along with their frequencies. During test time, if any misspelled word is found, then the program will be replacing that word with most appropriate word from that knowledge-base.
But here is a problem. As I consider all the tokens along with there frequencies, then there will be some rare words as well which are not too much helpful. Even in worse cases, they may prevent a word from being corrected with another more preferable option. Following is an example:
Token    Frequency
========================
quick      122
brown       99
fox         80
dog         79
.
.
.
quik         1

Here we can see that, the knowledge-base contains both quick and quik, so if the input contains the term quik then it will not be corrected as it is already present in the knowledge-base. But leaving it as it is (just because it is an in-domain token) is not a good idea either. As, it is obvious that, it would be better if there was no quik in the knowledge-base such that the term got a chance to be corrected with quick. As quick is more familiar to the text classifier.
So, the obvious way is to remove the tokens with low frequencies. But I am facing problem to choose a threshold value of this frequency (i.e, choosing a threshold value such that if some token has lower or equal frequency than that, then we will call them as rare words and remove from the knowledge-base). Deciding this value is problematic, as it can vary a lot based on the data-set we are using. For instance, for a large data-set we might have chosen this value to be 3 which is reasonable. But for small data-set this value will be too much and many important tokens will be removed.
Following is another view of the same problem: In this plot I have plotted the first 1000 most frequent tokens along with there frequencies (the total number of tokens is around 40,000). So, basically, I need a threshold for the frequency, under or equal of which I shall discard all the tokens, assuming them as rare words. Please note that, the data-set I have used to collect tokens is large (around 70,000 sentences). For smaller data-sets we may have different types of frequency distributions. So, I am suspecting that, the presupposition on the distribution might not be appropriate for all the cases, therefore, we might need to use rank-statistics.

So, the question is is there any statistical / mathematical approach to find this threshold dynamically?
 A: I'll give my two cents, though it's far from a "perfect" solution (if one exists).
First, as for the issue of varying dataset sizes, you can normalize the token frequencies by the document count, and choose a threshold over the normalized frequency rather than the absolute one.
Second, as you noticed, using a threshold in such cases is problematic, and will always create false positives and/or false negatives. One thing you could do is sample rare words across various frequencies, manually label them as typos or legit rare words, choose a metric for weighing between false positives and false negatives, and use all that to find a threshold that optimizes that metric. This will still not be perfect, 1) because frequency alone probably can't perfectly discriminate typos from legitimately rare words, and 2) because it will be based on manual labeling of samples (and not all the data).
Looking outside the dataset, you can perhaps intersect your knowledge base with existing language databases, e.g. wordnet or others in different languages.
Lastly, if you're up to some more sophisticated cleaning work, notice that you can use more information than just frequency to try determine if a word is a typo or not: say we have two words with identical low frequencies, e.g. quik and Cthulu. quik is very similar (e.g. using edit distance) to an existing, popular word in the dataset (quick), which increases its likelihood of being a typo; while Cthulu is probably distant from most other words in the dataset. This way you can use heuristics to assign a "typo probability" to each word, and use a cutoff on that score.
