Understanding the process of transfer learning for NLP

Question

Full Disclosure: I am a machine learning newbie.

I have been learning about natural language processing for the past few weeks. To my understanding, the process of creating a supervised text model works as follows:

1. Find a labeled corpus
2. Represent entries from this corpus as a vector using a manor of techniques (one hot encoding, n-grams, TF-IDF etc.).
3. Train some model on the processed vectors.

tl;dr: What off the shelf models are available for step 3, and how do I "update" these models for my corpus?

I want to use a supervised model for sentiment analysis, however my labeled corpus is small (only around 1000 entries).

With a simple approach using tf-idf, bigrams, and stemming for step 2 and scikit-learn's LinerSVC for step 3, I achieved an accuracy of 74.77+-11.05 sorting into 3 discrete classes: positive, indifferent, negative.

I now want to use transfer learning, partially fitting a pre-trained model on my corpus for better accuracy. The idea here is my corpus is based on a subset of all text, therefore it has its own jargon, however much of sentiment from other contexts will be "transferable."

For example, an off the shelf model trained on millions product reviews like: "My iPhone broke for the 8th time, I have nothing but negative feelings for this hunk of junk." could be updated with relatively few examples to classify: "markets were bearish today amongst negative analyst estimates".

I have read quite a few articles, and it seems like there are plenty of models to use transfer learning for step 2 of this problem. To my understanding, BERT is one of these. At a high level, BERT is trained on a massive corpus to plot words in a vector space based upon their similarity to other words, so a representation of the strings ["cat","dog"] may be [0.654, 0.682] as both these words are nouns representing furry animals, while ["cat", "running"] may be [0.654, 0.322].

I understand how BERT can be used as a transfer learning solution to step 2 of this problem. What I don't understand is what off the shelf models are available for step 3, and how to "update" these models for my corpus.

Most posts I've read on transfer learning for NLP spend a lot of time on step 2 of this problem and breeze over step 3 by training a DNN on a small BERT-processed corpus. This confuses me as to my understanding DNNs require very large datasets to perform. Is this not the case? I feel as though I am missing this important step 3 in transfer learning for NLP.

Answer 1

"Transfer learning" is not really straight forward. For example, you can reuse your encodings such n-grams. This is done by making the dictionary of your previous encoder the vocabulary input for your new encoding. You can see however, that for TF-IDF, if you add more sentences it will change the frequency of words in the corpus. Reusing the vocabulary makes a TF-IDF no longer a true TF-IDF. The same is the case for linear or logistic regression. If you add new data, the standard deviation, standard errors, and mean has to be recalculated for every point.

All this is to say, that if you want perform transfer learning use a deep learning model such as a recurrent neural net or LSTM.

If you want to use machine learning tools, merge your data with the new data set, re-encode your data and rerun your pipeline from scratch.