0
$\begingroup$

I have a question. There's a library (uses this paper) which suggests in its cross lingual part that if the XLM-R is trained in english dataset, it can be directly applied to datasets in other languages, and zero shot cross lingual transfer can be conducted.

So, my question is, if I trained XLM-R for english summarisation task, will it be able to transfer that knowledge and generate summaries in other languages using zero shot cross lingual transfer? I already have code written and tested for small dataset, but it requires a good amount of computational power for the whole dataset, so that's the reason for asking this question.

Edit: A little update for anyone who was interested in this question. I tried to train the XLM-R on a very little portion of food review dataset (5500 examples), it still seems to output some summaries (1-2 words on test dataset). This was for english language. I switched the language to german (I think it's lexically similar to english. I might be wrong as neither of the languages are my first language, anyways), but didn't train the model on the german dataset, and the output was horrible. It's giving the same words as the summaries. I even switched the language to japanese for the test dataset alone, still no results were found (as expected).

Improve this question
$\endgroup$
2
  • $\begingroup$ Do you mean abstractive (i.e., generating a coherent textual summary) or extractive summarization (i.e., highlighting several sentences that best capture the content)? $\endgroup$
    – Jindřich
    Commented Oct 19, 2022 at 7:47
  • $\begingroup$ Hi, I'm talking about abstractive summarisation here. I found a blog on huggingface that was working with RoBERTa, so I adapted it for XLM-R. I would still like to know if the zero shot cross lingual transfer will work or not, because I haven't trained it on good amount of data. Thanks! $\endgroup$
    – lazytux
    Commented Oct 19, 2022 at 12:02

1 Answer 1

Reset to default
0
$\begingroup$

Cross-lingual transfer for text generation is a much more difficult task that cross-lingual transfer for classification. In classification, you hope that similar input sentences will get a similar representation regardless of the language, and most importantly the target labels are shared.

In the summarization setup that you describe, the target labels are different. Similar inputs will get similar representations in different languages, but you suddenly want the decoder to generate different outputs for them. It is destined to fail.

Such a setup might be interesting for anything-to-English summarization. There are similar experiments in machine translation where they use XLM-R a universal encoder, to train translation from several languages into English and what they get is a translation from many languages into English. It, however, requires a smart training scheme to prevent catastrophic forgetting, so that XLM-R keeps the knowledge about all languages.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks, yes I understood the message you want to convey. I read about catastrophic forgetting, and the results I got established the fact that model forgot about other languages when it was fine-tuned for the english language dataset (apologies if I said something wrong). I haven't done the experiments for the machine translation part yet though. $\endgroup$
    – lazytux
    Commented Oct 24, 2022 at 11:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.