I have taken a corpus of English for finding the word frequency in giving recommendation for spelling mistakes.

I just used simple : Edit Distance between recommended and misspelled word (ED_rw) and frequency of recommended word (F_rw) with words sorted by :

    ED_rw / F_rw

Now I am required to give suggestions for other languages. Finding edit-distance is still the same. However, how do I get the corpus for other languages?

I think that simple localization of corpus should also work (translation of the text, preserving its semantic meaning). The corpus contains texts from novels etc.. and represents the general talk that we do in life.

Statistically, I only use corpus for the frequency of words in that language. Every language would have its own set of these variation in usage of words. This variation is guaranteed to be captured in any general text of that language. Hence, localization should also work. I admit that this localization should preserve the semantics of the text.

Is my thinking right ?

Would a localized version of general text of English be good for finding word frequencies in other languages ?

  • $\begingroup$ Welcome to Cross Validated! What exactly does "localization" of an English text mean in this context? $\endgroup$ – Scortchi - Reinstate Monica Apr 2 '15 at 11:50
  • 1
    $\begingroup$ @Scortchi thanks.. by localization i meant translation of text and preserving semantic meaning.. $\endgroup$ – Ashish Negi Apr 2 '15 at 12:12
  • $\begingroup$ Thanks - I've taken the liberty of adding that definition to your question. $\endgroup$ – Scortchi - Reinstate Monica Apr 2 '15 at 12:14

I think translation could produce results close to original word frequencies, assuming expiriencing human translator. However, this will be subject to two kinds of error: first, word usage will reflect translator own preferences for word choices, (this can theoretically be fixed by using a number of different translations) and second, seeing original English sentences can also affect word choices and phrase choices. For some words, such as people names, organization titles, locations, and other similar words, results will probably be unsatisfactory, but for less specific words, results should be reasonable good approximation.

That said, I don't see the reason to use this approach for except for rare and dead languages. It is costly and time consuming task to translate enough text. For many languages, reasonable corpus of native language speakers writings can be obtained at a small fraction of a cost, for example by downloading language-specific wikipedia pages (they are avaliable to download as archives), getting some research corpus, developed for that language, or at last resort by crawling some portion of relevant web pages.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.