I have taken a corpus of English for finding the word frequency in giving recommendation for spelling mistakes.
I just used simple :
Edit Distance between recommended and misspelled word (ED_rw) and
frequency of recommended word (F_rw) with words sorted by :
ED_rw / F_rw
Now I am required to give suggestions for other languages. Finding edit-distance is still the same. However, how do I get the corpus for other languages?
I think that simple localization of corpus should also work (translation of the text, preserving its semantic meaning). The corpus contains texts from novels etc.. and represents the general talk that we do in life.
Statistically, I only use corpus for the frequency of words in that language. Every language would have its own set of these variation in usage of words. This variation is guaranteed to be captured in any general text of that language. Hence, localization should also work. I admit that this localization should preserve the semantics of the text.
Is my thinking right ?
Would a localized version of general text of English be good for finding word frequencies in other languages ?