I'm considering the topic of generating an NLP Embedding for a language without a written standard or a significant corpus. I realized that as challenging as that is, it is still not as challenging as generating and embedding for a language that doesn't have a written representation at all, which is probably the case for most of the world's languages (in terms of the number of languages, not number of speakers).
What are the approaches for generating an embedding for a language without a written representation at all? The following approach seems feasible:
- Record enough samples to capture all the phonemes in the language.
- Generate a suitable encoding of the phonemes.
- Record a large amount of speech in that language using the encoding from the previous step to convert it to a digital format.
- Use the data to generate an unsupervised embedding. Use this as the starting point for downstream classification and more complex NLP tasks.
This seams perfectly feasible, but I feel I might be missing something:
- Most languages don't have a one-to-one mapping between sequences of phonemes and sequences of symbols, e.g. homographs or homophones.
- Is it as straightforward as I think it is to capture all the unitary phonemes from a sufficiently large sample? More specifically, this might be informed by my familiarity with English, Arabic, French (alphabet based), and Japanese (diphtongs) - all with a relatively small and discrete set of phonemes. But this might not be the case with other languages with more complex pronunciation rules, like tonal languages, or subsaharan African languages?
- Some languages have demographic based pronunciation variations (based on gender, social class, etc...). Would these variations make a direct phoneme-to-binary encoding more challenging?
To clarify where I am coming from, based on Tim's response:
- For the actual problem at hand, yes the data is available. It is a local dialect (Tunisian Arabic) of a standard language that already has a wide corpus and some embeddings available (Modern Standard Arabic). It is however different enough from MSA, as to be mutually unintelligible, and have significant variations in grammar, syntax, cases, etc.... Tunisian Arabic does have a large body of streaming and older video data (and a lot of it captioned into English or French actually).
- I originally went down the path of trying to figure out whether the embeddings from the more standard parent language were useful, maybe they could be fine tuned to Tunisian Arabic. I assumed somebody had already researched the relevance of Mandarin embeddings for Cantonese NLP problems, or Spanish embeddings for Catalan and Andalusian, etc...
- But then how do we solve for the lack of written standard, and the fact that different demographics use different alphabets for the task, either a modified Arabic alphabet to account for additional sounds not present in MSA, or latin letters augmented with numbers to account for both native MSA sounds and Tunisian specific sounds (e.g. the proper spelling of "Ali" is actually "3ali" using this more recent informal norm.
- Then I realized that it might be easier to just treat it as a language with no written representation at all, since that problem might have been researched more often than the somewhat complex dialect type problem I just mentioned. Hence my post.