0
$\begingroup$

i am developing a text classification neural network based on this two articles - https://github.com/jiegzhan/multi-class-text-classification-cnn-rnn https://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/

For the training i am using, text data in Russian language (language essentially doesn't matter,because text contains a lot of special professional terms, and sadly to employ existing word2vec won't be an option.)

I have such parameters of training data - Maximum lengths of an article - 969 words Size of vocabulary - 53886 Amount of labels - 12 (sadly they are distributed quite unevenly, for instance i have first label - and have around 5000 examples of this, and second contains only 1500 examples.)

Amount of training data set - Only 9876 entries. I'ts the biggest problem, because sadly i can't increase size of the training set by any means (only way out to wait another year☻, but even it will only make twice the size of training date, and even double amount is'not enough)

Here is my code -

x, x_test, y, y_test = train_test_split(x_, y_, test_size=0.1)
x_train, x_dev, y_train, y_dev = train_test_split(x, y, test_size=0.1)

embedding_vecor_length = 100

model = Sequential()
model.add(Embedding(top_words, embedding_vecor_length, input_length=max_review_length))
model.add(Conv1D(filters=32, kernel_size=3, padding='same', activation='relu'))
model.add(MaxPooling1D(pool_size=2))
model.add(keras.layers.Dropout(0.3))
model.add(Conv1D(filters=32, kernel_size=4, padding='same', activation='relu'))
model.add(MaxPooling1D(pool_size=2))
model.add(keras.layers.Dropout(0.3))
model.add(Conv1D(filters=32, kernel_size=5, padding='same', activation='relu'))
model.add(MaxPooling1D(pool_size=2))
model.add(keras.layers.Dropout(0.3))
model.add(Conv1D(filters=32, kernel_size=7, padding='same', activation='relu'))
model.add(MaxPooling1D(pool_size=2))
model.add(keras.layers.Dropout(0.3))
model.add(Conv1D(filters=32, kernel_size=9, padding='same', activation='relu'))
model.add(MaxPooling1D(pool_size=2))
model.add(keras.layers.Dropout(0.3))
model.add(Conv1D(filters=32, kernel_size=12, padding='same', activation='relu'))
model.add(MaxPooling1D(pool_size=2))
model.add(keras.layers.Dropout(0.3))
model.add(Conv1D(filters=32, kernel_size=15, padding='same', activation='relu'))
model.add(MaxPooling1D(pool_size=2))
model.add(keras.layers.Dropout(0.3))
model.add(LSTM(200,dropout=0.3, recurrent_dropout=0.3))
model.add(Dense(labels_count, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

print(model.summary())

model.fit(x_train, y_train, epochs=25, batch_size=30)
scores = model.evaluate(x_, y_)

I tried different parameters and it gets really high accuracy in training (up to 98%) But i really performs badly on test set. Maximum that i managed to achieve was around 74%, usual result something around 64% And the best result was achieved with small embedding_vecor_length and small batch_size.

I know - that my test set is only 10 percent of training test, and overall data-set is the biggest problem, but i want to find a way around this problem.

So my questions are - 1) Is it correctly builded model for text classification purpose? (it works) Do i need to use simultaneous convolution an merge results instead? I just don't get how the text information doesn't get lost in the process of convolution with different filter sized (like in my example) Can you explain hot the convolution works with text data? There are mainly articles about image recognition..

2)i obliviously got a problem with overfitting my model. How can i make the performance better? I have already added Dropout layers. What can i do next?

3)May be i need something different? I mean pure RNN without convolution?

$\endgroup$
0
$\begingroup$

For your 10k examples, this model is way too complex in my opinion. I don't know your task specifically, but think about whether you really need that many "layers of abstraction" to determine if the text belongs in one class or another.

As a general guideline, I would start with a very simple model (i.e. throw out all those Conv-Pool-Dropout-Combos except for the very first). Train that. If the training error is low, it means the model is not complex enough to learn the problem, so you should increase its size (either with more layers or more neurons/filters per layer. Or possibly you have to change the types of layers/their configuration). If the test error is much lower than your training error, that means your model is learning the training data by heart. Then you should add more regularization and/or make the model smaller (removing layers or decreasing layer size).

You can also try pure RNNs (I would recommend GRUs), but note that their training takes much more time.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.