0
$\begingroup$

I am learning using Keras Embedding layer to build embedding models. However, I failed to build a good embedding model. Can anyone help me check where I did wrong? Or not enough data to train?

Data source is 20newgroups from sklearn (sample size=11314). It is a 20-class classification problem.

I followed the a Towards Data Science post: Machine Learning — Multiclass Classification with Imbalanced Dataset.

Text data preprocess

The code to preprocess the text data is follows

from sklearn.datasets import fetch_20newsgroups
from sklearn.preprocessing import LabelBinarizer
from keras.preprocessing.text import Tokenizer

train, test = fetch_20newsgroups(subset='train'), fetch_20newsgroups(subset='test')

tokenizer = Tokenizer()
tokenizer.fit_on_texts(train.data)

vocab_size=len(tokenizer.word_index) + 1
max_length = max([len(s.split()) for s in train.data])
print ("max_length:", max_length)

X_train_tokens = tokenizer.texts_to_sequences(train.data)
X_test_tokens = tokenizer.texts_to_sequences(test.data)

X_train_pad = pad_sequences(X_train_tokens, maxlen=max_length, padding='post')
X_test_pad = pad_sequences(X_test_tokens, maxlen=max_length, padding='post')

encoder = LabelBinarizer()
encoder.fit(train.target)
y_train, y_test = encoder.transform(train.target), encoder.transform(test.target)

Now each data point in X_train_pad its represented as

array([[   14, 32671,  4308, ...,     0,     0,     0],
       [   14, 14824,  3321, ...,     0,     0,     0]], dtype=int32)

each data point is padded to have 11821 features. I tried two classifier models.

Embedding + GRU

First, I train a classifier model using Keras' Embedding layer, GRU layer:

from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout, Flatten

embedding_dim= 100
model = Sequential()
model.add(Embedding(input_dim=vocab_size, output_dim=embedding_dim, input_length=max_length))
model.add(GRU(units=32, dropout=0.2, recurrent_dropout=0.1))
model.add(Dense(num_labels, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

history = model.fit(X_train_pad, y_train, batch_size=128, epochs=10, verbose=2, validation_split=0.2)

which showed terrible performance

Train on 9051 samples, validate on 2263 samples
Epoch 1/20
 - 1074s - loss: 2.9925 - acc: 0.0530 - val_loss: 2.9950 - val_acc: 0.0477
Epoch 2/20
 - 1098s - loss: 2.9904 - acc: 0.0497 - val_loss: 2.9946 - val_acc: 0.0482
Epoch 3/20
 - 1015s - loss: 2.9902 - acc: 0.0529 - val_loss: 2.9948 - val_acc: 0.0486
Epoch 4/20
 - 1029s - loss: 2.9899 - acc: 0.0504 - val_loss: 2.9963 - val_acc: 0.0486
Epoch 5/20
 - 1073s - loss: 2.9897 - acc: 0.0531 - val_loss: 2.9955 - val_acc: 0.0464
Epoch 6/20
 - 1024s - loss: 2.9895 - acc: 0.0554 - val_loss: 2.9956 - val_acc: 0.0464
Epoch 7/20
 - 1084s - loss: 2.9893 - acc: 0.0526 - val_loss: 2.9954 - val_acc: 0.0495
Epoch 8/20
 - 1090s - loss: 2.9888 - acc: 0.0537 - val_loss: 2.9945 - val_acc: 0.0464
Epoch 9/20
 - 1112s - loss: 2.9882 - acc: 0.0535 - val_loss: 2.9942 - val_acc: 0.0521
Epoch 10/20
 - 1088s - loss: 2.9871 - acc: 0.0528 - val_loss: 2.9920 - val_acc: 0.0464

Regular NN model

In comparison, rather than using the Embedding layer, convert input data with tdidf score

X_train = tokenizer.texts_to_matrix(train.data, mode='tfidf')
X_test = tokenizer.texts_to_matrix(test.data, mode='tfidf')

and use the following model:

model = Sequential()
model.add(Dense(512, input_shape=(vocab_size,)))
model.add(Activation('relu'))
model.add(Dropout(0.3))
model.add(Dense(512))
model.add(Activation('relu'))
model.add(Dropout(0.3))
model.add(Dense(num_labels))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

history = model.fit(X_train, y_train, batch_size=128, epochs=10, verbose=2, validation_split=0.2)

the result is much better:

Train on 9051 samples, validate on 2263 samples
Epoch 1/10
 - 116s - loss: 0.8293 - acc: 0.7971 - val_loss: 0.3054 - val_acc: 0.9275
Epoch 2/10
 - 114s - loss: 0.0748 - acc: 0.9882 - val_loss: 0.3221 - val_acc: 0.9205
Epoch 3/10
 - 112s - loss: 0.0459 - acc: 0.9952 - val_loss: 0.3438 - val_acc: 0.9169
Epoch 4/10
 - 112s - loss: 0.0415 - acc: 0.9951 - val_loss: 0.3681 - val_acc: 0.9174
Epoch 5/10
 - 112s - loss: 0.0636 - acc: 0.9933 - val_loss: 0.4106 - val_acc: 0.9152
Epoch 6/10
 - 113s - loss: 0.0344 - acc: 0.9964 - val_loss: 0.4753 - val_acc: 0.9015
Epoch 7/10
 - 112s - loss: 0.0369 - acc: 0.9962 - val_loss: 0.4233 - val_acc: 0.9134
Epoch 8/10
 - 112s - loss: 0.0460 - acc: 0.9959 - val_loss: 0.4342 - val_acc: 0.9099
Epoch 9/10
 - 113s - loss: 0.0350 - acc: 0.9966 - val_loss: 0.4276 - val_acc: 0.9143
Epoch 10/10
 - 112s - loss: 0.0298 - acc: 0.9967 - val_loss: 0.4931 - val_acc: 0.9050

Does anyone play this data before? Is there a bug or data amount 11k is too small to build embedding model?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.