Generating embeddings for languages without a written representation?

Question

I'm considering the topic of generating an NLP Embedding for a language without a written standard or a significant corpus. I realized that as challenging as that is, it is still not as challenging as generating and embedding for a language that doesn't have a written representation at all, which is probably the case for most of the world's languages (in terms of the number of languages, not number of speakers).

What are the approaches for generating an embedding for a language without a written representation at all? The following approach seems feasible:

• Record enough samples to capture all the phonemes in the language.
• Generate a suitable encoding of the phonemes.
• Record a large amount of speech in that language using the encoding from the previous step to convert it to a digital format.
• Use the data to generate an unsupervised embedding. Use this as the starting point for downstream classification and more complex NLP tasks.

This seams perfectly feasible, but I feel I might be missing something:

• Most languages don't have a one-to-one mapping between sequences of phonemes and sequences of symbols, e.g. homographs or homophones.
• Is it as straightforward as I think it is to capture all the unitary phonemes from a sufficiently large sample? More specifically, this might be informed by my familiarity with English, Arabic, French (alphabet based), and Japanese (diphtongs) - all with a relatively small and discrete set of phonemes. But this might not be the case with other languages with more complex pronunciation rules, like tonal languages, or subsaharan African languages?
• Some languages have demographic based pronunciation variations (based on gender, social class, etc...). Would these variations make a direct phoneme-to-binary encoding more challenging?

---

To clarify where I am coming from, based on Tim's response:

• For the actual problem at hand, yes the data is available. It is a local dialect (Tunisian Arabic) of a standard language that already has a wide corpus and some embeddings available (Modern Standard Arabic). It is however different enough from MSA, as to be mutually unintelligible, and have significant variations in grammar, syntax, cases, etc.... Tunisian Arabic does have a large body of streaming and older video data (and a lot of it captioned into English or French actually).
• I originally went down the path of trying to figure out whether the embeddings from the more standard parent language were useful, maybe they could be fine tuned to Tunisian Arabic. I assumed somebody had already researched the relevance of Mandarin embeddings for Cantonese NLP problems, or Spanish embeddings for Catalan and Andalusian, etc...
• But then how do we solve for the lack of written standard, and the fact that different demographics use different alphabets for the task, either a modified Arabic alphabet to account for additional sounds not present in MSA, or latin letters augmented with numbers to account for both native MSA sounds and Tunisian specific sounds (e.g. the proper spelling of "Ali" is actually "3ali" using this more recent informal norm.
• Then I realized that it might be easier to just treat it as a language with no written representation at all, since that problem might have been researched more often than the somewhat complex dialect type problem I just mentioned. Hence my post.

Answer 1

Embeddings, as other neutral networks, need huge datasets. People usually scrape thousands of books, news articles and web pages for that. For audio files, this would mean thousands of hours of recordings, considering audio quality issues, probably even more. For English, this would mean something like scraping all the audio from Netflix etc. Are you sure you have this data? If not, recording it sounds like a project for years.

Moreover, embeddings are usually trained in scenarios like “guess next word”, so the data would be manually labeled by someone to mark beginnings and ends of words. This alone is also a lot of work.

The reasons above is why we have a lot of NLP models for English and not many for other languages. It is easy to scrape the Internet for thousands of documents in English. Same for written language: you don’t have audio quality issues to worry about, written language is already recorded in unified form, words are easily separable in most languages etc. With audio there’s much more additional preprocessing needed.

You are correct about the other issues mentioned in bullet points by you. You would probably need samples diverse in terms of regional variations of pronunciation, genders, age, education, etc. to make the data representative.