Full Disclosure: I am a machine learning newbie.
I have been learning about natural language processing for the past few weeks. To my understanding, the process of creating a supervised text model works as follows:
- Find a labeled corpus
- Represent entries from this corpus as a vector using a manor of techniques (one hot encoding, n-grams, TF-IDF etc.).
- Train some model on the processed vectors.
tl;dr: What off the shelf models are available for step 3
, and how do I "update" these models for my corpus?
I want to use a supervised model for sentiment analysis, however my labeled corpus is small (only around 1000 entries).
With a simple approach using tf-idf, bigrams, and stemming
for step 2
and scikit-learn's LinerSVC
for step 3
, I achieved an accuracy of 74.77+-11.05
sorting into 3 discrete classes: positive, indifferent, negative
.
I now want to use transfer learning, partially fitting a pre-trained model on my corpus for better accuracy. The idea here is my corpus is based on a subset of all text, therefore it has its own jargon, however much of sentiment from other contexts will be "transferable."
For example, an off the shelf model trained on millions product reviews like: "My iPhone broke for the 8th time, I have nothing but negative feelings for this hunk of junk."
could be updated with relatively few examples to classify: "markets were bearish today amongst negative analyst estimates"
.
I have read quite a few articles, and it seems like there are plenty of models to use transfer learning for step 2
of this problem. To my understanding, BERT is one of these. At a high level, BERT is trained on a massive corpus to plot words in a vector space based upon their similarity to other words, so a representation of the strings ["cat","dog"]
may be [0.654, 0.682]
as both these words are nouns representing furry animals, while ["cat", "running"]
may be [0.654, 0.322]
.
I understand how BERT can be used as a transfer learning solution to step 2
of this problem. What I don't understand is what off the shelf models are available for step 3
, and how to "update" these models for my corpus.
Most posts I've read on transfer learning for NLP spend a lot of time on step 2
of this problem and breeze over step 3
by training a DNN on a small BERT-processed corpus. This confuses me as to my understanding DNNs require very large datasets to perform. Is this not the case? I feel as though I am missing this important step 3
in transfer learning for NLP.