Text classification with small dataset for a specialized domain

Question

I have a multiclass text classification problem where I have very few documents for each class. The classes are imbalanced but I want to be able to predict the class when I have at least 200 - 300 documents. What methods do you suggest? Also the documents are from a specialized domain so I dont think transfer learning will work. Would you recommend word embeddings when working with specialized domain text?

Answer 1

I would run a sequential neural network in python with keras: https://keras.io/getting-started/sequential-model-guide/

The idea is to vectorize your textes first with the words. For example:

I have a dream about yesterday
I want to sleep

Would give you the Words list: 'I, have, a, dream, about, yesterday, want, to, sleep' Then this words will be your features:

  I have a dream about yesterday want to sleep
1 1 1    1   1    1     1         0   0   0
2 1 0    0   0    0     0         1   1   1 

with your target y:

  classA classB
1  1      0
2  0      1

then you can remove all the stop words with some techniques, to reduce the amount of vectors in your algorithm, and then run the neural network. In the end you will have a big vector for every know class, where every word gets a number of occurence in that class, with a lot of zeros.

This will be memory expensive, but quite succesful.

Answer 2

I would always recommend you use the pre-trained variables trained using a large corpus because those pretrained parameters are kind of like prior and regularization.

Even if your task is very specific the low-level feature extraction or pattern learning can most probably be shared. To prevent some pitfalls you can fix some parameters, like the embedding layer(or and some lower layers). Because most of the embeddings will not be retrained and only the words in your specific task will be retrained those un-retrained embeddings may conflict with the retrained ones.