need guidance for deep learning NLP project

Question

For a personal project, I need to build a deep learning model that can accomplish the following task:

I have a small corpus of english texts. for each text, there are 2 versions - one regular in plain english, and one that is the same except that someone added one of seven "new" words in between regular english words, with some unknown logic behind it.

so, for example, the sentence: "the fast cat ran away from the brown dog" can become "the fast WORD1 cat ran away WORD2 from the brown dog WORD1"

there are only 7 of those new words.

I want to train a model that can learn the logic behind the insertions and apply it to new texts, so they will have those words, placed by the same logic.

I have several ideas:

1. organise this as a translation problem, from english to english with 7 more words in vocab. the challenge here is that there are no pre-trained model that I know about the translate from english to english, and my corpus is too small to train from scratch. also, I need all of the words from the original sentence to stay the same, and translation might replace them with similar words.

2. have the model go word by word in the original sentence and for each step, predict if the next word in the original should come next or one of the 7 new words.

what do you think about those ideas? do you have a better one? what pre-trained model might be able to be a good starting point?

thank you.

Answer 1

You can treat this problem as a code-mixing problem. Code-mixing is the mixing of two or more languages or language varieties in speech [A]. There have been some effort in generating code-mixed data [B] and it might be interesting, if not helpful, to look through them and see if they directly apply to your problem.

[A] Wikipedia - Code Mixing
[B] Language Modeling for Code-Mixing: The Role of Linguistic Theory Based Synthetic Data, ACL 2018