1
$\begingroup$

I have 2 lists of word2vec vectors. First list has vectors, which represent fruits names, second list has vectors which represent vegetables names.

I am trying to train neural network on this 2 lists. Output for fruits is [1, 0] and output for vegetables is [0, 1]

Finally, when network will be trained, I want to feed word2vec vector of word "fruits" and receive output [1, 0]. And for "vegetables" I want receive output [0, 1].

But I have trouble: when network is training, accuracy index "freeze" at value 0.5

I use pre-trained "lexvec.enwiki+newscrawl.300d.W.pos.vectors" word2vec model with gensim, keras with Dense layers.

Here is NN model:

model = Sequential()
model.add(Dense(50, input_dim=300, activation='relu'))
model.add(Dense(400, activation='relu'))
model.add(Dense(400, activation='relu'))
model.add(Dense(2, activation='softmax'))
model.compile(optimizer='sgd', loss='mse', metrics=["accuracy"])

What should I do to make my classification work?

$\endgroup$
1
$\begingroup$

The first thing to consider is the size of your training data. From your description and results I gather that it is small. There are three things your need to:

  1. Change your loss to categorical crossentropy and use a single output since you have boolean classes.

  2. Significantly decrease the size of your network. First try a single hidden layer with far fewer hidden units.

  3. Split your data into training, validation, and test sets. If your dataset is small, use cross-fold validation.

I would recommend starting with this code (logistic regression):

model = Sequential()
model.add(Dense(1, input_dim=300, activation='sigmoid'))
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=["accuracy"])

If that fails, add a hidden layer:

model = Sequential()
H = 100
model.add(Dense(H, input_dim=300, activation='tanh'))
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=["accuracy"])

H is the hyper-parameter you'll have to tune. Try 50, 100, 150, 200, 300.

Now if your classes aren't balanced (not 50/50 fruits/veggies) you should look at other metrics such as precision, recall, and F1.

Good luck!

$\endgroup$
  • $\begingroup$ thanks for answer. I've already solved my issue. The problem was that I forgot to remove pad_sequences call from my copypasted code. $\endgroup$ – Dmitry Nalyvaiko Mar 18 '17 at 23:28
  • $\begingroup$ Please heed this advice though as it should improve your results. $\endgroup$ – Alexandre Mar 18 '17 at 23:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.