Text Embeddings on a Small Dataset

Question

I am trying to solve a binary text classification problem of academic text in a niche domain (Generative vs Cognitive Linguistics). My target text data consists of near 400 paper abstracts with less than 300 words in each. Previously I tried to use Doc2Vec in order to solve the problem, but the best accuracy that I could get was around 82%. I have since tried to use pre-trained vectors but the consensus on Doc2Vec is that it is best not to use pre-trained doc2vecs. I have tried using pre-trained Word2Vec models, but the models are usually huge and my laptop (8 GBs of RAM) cannot handle loading them. So as a result I tried to collect a larger source data myself, and train a word embedding model on that data, and then use the word vectors in the target domain.

I have collected more than 70K of paper abstracts in the related fields (mostly papers categorized with Linguistics tag), and have trained FastText, Doc2Vec and Word2Vec models on the source data. But after using these models in the target domain, the results are not even better than my previous attempts with a simple Doc2Vec, let alone being marginally better.

I have also tried using TFIDF and CountVectorizer on the target domain, but the results do not become better.

Yesterday I stumbled upon this implementation of getting document vectors using Word2Vec and TF IDF simultaneously, but to my surprise, the results are on par with averaging the documents.

I was thinking of maybe using active learning in the process? Since maybe the bottleneck is the very small target dataset. Or maybe generating synthetic texts similar to the target data?

Thank you for reading this.

Answer 1

For this type of problem you may want to consider ULMFiT or a similar fine-tuning approach. The advantage of this approach is that you fine-tune a language model trained on another corpus to your smaller corpus (of ~70k) and then you create a classifier on-top of a whole language model on the small amount of labelled data. I.e. instead of just using a first-layer embedding like the ones you mentioned, you get to use structure in the whole neural network (that you can fine-tuning with appropriate careful unfreezing of layers). This is implemented e.g. in the fastai python library and described in the lectures of the excellent fast.ai deep-learning course. Additionally, you may consider some data augmentation during your training (e.g. round-trip translation or word/phrase-substitution).

Answer 2

The approach of averaging vectors helped me get 0.90 auc with random forest in the binary text classification problem. It is easy to understand and painless to implement. The idea is the following:

1. Load the pre-trained word2vec model: https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit (1.5GB).
2. Draw vectors from the loaded word2vec model for each word in the document (sentence/abstract).
3. Aggregate (average) the vectors for each document: sum all word vectors and divide by the number of words in a document.

You can check the following code for the implementation of the approach:

1. load the word2vec model

2. make_agg_vec

Answer 3

So after a few weeks of experimenting, I have reached a new solution. Since I had suspected that the bottleneck of my problem was probably my small target dataset, I tried to use data augmentation to increase the data. I tried simple shuffling and insertion of random words, but it didn't really help me with this problem since my problem is very domain specific, so I tried to use generative models in order to generate texts similar to what I have in the target dataset. I used the awesome textgenrnn library and trained two different models for my two classes, and generated 1000 texts of 300 words long for each class.

Then I used these generated texts alongside a portion of my original data to train a linearSVC, on the features I extracted with the TF-IDF * WordEmbedding approach that I have mentioned in my original post, and tested the system only on the original data. The results were improved by 5% and I got 89% accuracy.

The next steps will be to train the text generator on a bigger dataset (that 70k dataset for example) and then finetune it for a few epochs on the target data.

Also the approach that björn suggests sounds really interesting, I shall experiment with that too.

Answer 4

If your options are not limited to some word2vec based approach I suggest using topic modelling as a method to map document to a low dimensional vector.

In my experience for 400 documents with less than 300 words you could expect to extract 3-5 somehow meaningful topics using Latent Dirichlet allocation model. Training topic model with 70K abstracts dataset and 25-50 topics could also be fruitful.

Gensim implements LDA for Python and there are a lot of tutotials on the internet. You could also give a try to BigARTM.