I am currently training an mt5 in Spanish to English (and vice versa) translation. That works with a bleu of 40ish. (both ways). I then want to use that same model I trained to improve the translation of Spanish to a low-resource language.

The way I'm doing it, I go through the whole translation process and then evaluate. After that I reload the model (since I'm doing this on Google Colab and by then it has nearly timed out so I need to restart the runtime) by doing:

model = AutoModel.from_pretrained("/content/drive/checkpoint-11076-epoch-1")

Then I try to train with the new data by doing:

model.train_model(train_df, eval_data=eval_df)

(with new train and eval data frames for the new languages), however, it won't train them for long, maybe a few seconds.

I know this can be done because I've read about people training in a language pair and then another, but maybe I'm missing something crucial?

My intent is to train on a language pair, then use that and train that same model with the lower-resource language (as far as I'm aware this is called fine-tunning transfer learning, am I wrong?)



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.