Do I need training data in multiple languages for a multilingual transformer?

Question

I am attempting to train a transformer which can categorize sentences into one of n categories. This model should be able to work with a number of different languages - English and Arabic in my case.

Do I need to have labelled training data in both English and Arabic to fine tune a pretrained transformer, such as BLOOM, or can I fine tune the model using only English samples, and then the model should also work well on Arabic samples for my classificaiton task, since the fine tuning only trained the classification head?

My thoughts are that the pretraining of this model should allow it to transform the same input texts in English and Arabic to the same (or similar) embedding, which the classification head would have learned to then predict these embeddings accurately through the fine tuning.

Answer 1

The multilingual pre-trained models are in principle capable of so-called zero-shot language transfer, i.e., after being fine-tuned in one language, it should work in other languages covered by the representation as well. This, however, depends largely on the task: transfer in sentiment analysis seems to be easy, for more challenging tasks such as hate speech detection or fact-checking, the results are still very poor (in mid-2022).

When you have at least some data for Arabic, it is always better to use them. If you have unlabeled Arabic data for the task, you can use some data augmentation techniques.

BLOOM is a generative model and it is huge. If you are interested in closed-class categorization (rather than a generation), you should consider using encoder-only models such as mBERT, XLM-R, or RemBERT, which are considerably smaller are suitable for classification.