0
$\begingroup$

I'm trying to understand why my NN doesn't predict at all. I am using an embedding layer from gensim into keras to make a binary classification of paragraphs of text (similar to twitter sentiment analysis).

Here's the code :

from keras.models import Sequential
from keras.layers import Flatten
from keras.layers import Dense

keras_model = Sequential()
keras_model.add(embedding) // Added the embedding layer from gensim
keras_model.add(Flatten())
keras_model.add(Dense(1, activation='softmax'))
keras_model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc'])

I'll try to describe in more details my attempts so far :

  1. Initially I've trained the model using a dataset consisting of ~220k samples and I had 92.85% accuracy, which was great , but then I noticed that the ratio between negative and positive samples was exactly 0.928, which meant I needed to clean my dataset.

2 .I made the dataset with 50/50 distribution of positive to negative samples (~26k samples) then I tried the same and got accuracy of 50%.

3.Played around with different activations (relu, softmax , sigmoid) - no change or it dropped to 0% accuracy.

4.Added an extra hidden layer - again no change.

5.Tried different batch sizes (6,32,128,1024) - no change.

keras_model.summary()

Layer (type)                 Output Shape              Param #   
=================================================================
embedding_2 (Embedding)      (None, 500, 100)          596200    
_________________________________________________________________
flatten_5 (Flatten)          (None, 50000)             0         
_________________________________________________________________
dense_5 (Dense)              (None, 1)                 50001     
=================================================================
Total params: 646,201
Trainable params: 50,001
Non-trainable params: 596,200
_________________________________________________________________

Is there anything that can be done to get some real accuracy from this neural netowork ? Am I doing something wrong or the dataset is small to have a neural network as a classifier.

$\endgroup$
2
$\begingroup$

You apply softmax activation function on the output layer with only one output neuron. Softmax has to sum to 1 over all output neurons, and since you have only one of them, that one neuron will always output 1. Switch softmax to sigmoid and you're good.

$\endgroup$
  • $\begingroup$ I did try sigmoid as described, but no luck.. $\endgroup$ – Veselin Ivanov Jan 31 '18 at 18:53
  • $\begingroup$ Also try LSTM/GRU layer instead of Dense, because it seems like fully-connected one is a very bad choice for this job. $\endgroup$ – Lugi Feb 13 '18 at 12:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.