3
$\begingroup$

I want to train an LSTM Network, since it accepts Sequences as Input, and I plan to instead of using the bag-of-words representation, I want to replace each word with it's semantic vector, provided by word2vec or GloVe for example.

I'm trying to use that on Keras, and as output, I only want K outputs, representing the K categories that the text should belong.

Here is what I'm trying to do, but I can't figure out how can I finish it:

dataset = pd.read_csv('/home/brunoalano/data.csv', header=None, names=["text", "category"])
word_vector_model = Word2Vec()
index_dict = { w:i for i,w in enumerate(word_vector_model.model.model.index2word) }

model = Sequential()
model.add(
  Embedding(
    output_dim=word_vector_model.model.dimension,
    input_dim=n_symbols,
    mask_zero=True,
    weights=[embedding_weights],
    input_length=input_length
  )
)
model.add(LSTM(word_vector_model.model.dimension))
model.add(Dropout(0.5)) # try 0.3
model.add(Dense(1, activation='sigmoid'))

# Compile the model
model.compile(loss='binary_crossentropy', optimizer='adam')

So, it's possible to use this model for my purpose?

$\endgroup$

closed as off-topic by Sycorax, Michael Chernick, COOLSerdash, jbowman, mdewey Jul 4 '18 at 15:51

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – Sycorax, Michael Chernick, COOLSerdash, jbowman, mdewey
If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ Hello Brunno, look here: karpathy.github.io/2015/05/21/rnn-effectiveness at first picture - you probably would like to implement 3rd example, aren't you? Of course you can do this in keras. You should set return_sequence to False at the end of your LSTM, then you can connect output to e.g. Dense, or use it just as output of neural network. $\endgroup$ – 404pio Apr 8 '16 at 18:54
  • 1
    $\begingroup$ This is one of the examples provided in the Keras Documentation article Getting started with the Keras Sequential model under the header "Sequence classification with LSTM". After you copy-paste the code, use a categorical loss function. $\endgroup$ – Sycorax Jan 8 '18 at 1:38
0
$\begingroup$

First, I would add a display of the model. After the model.compile(...) add:

model.summary()

This will give you a good picture of the data flow through the layers.

Second, I would move the Dropout(...) layer right after the Embedding layer. If you are going to randomly drop samples then do it as early as possible to avoid doing unnecessary training.

Then, your Dense(...) layer should use activation='softmax'

Lastly, your model.compile should specify loss='categorical_crossentropy' since you want classification to 14 categories (not two which is binary classification). I would also suggest you add a: metrics=['accuracy'] to the model.compile.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.