I'm definitely a beginner and trying to get a handle on things so I apologise in advance for possible slowness of understanding :)

Recently, I came across a problem where given some sequence of categories $X=(x_1, x_2, \cdots, x_n)$ you had to predict another sequence of categories $y=(y_1, y_2, \cdots, y_n)$. Now, the future $X$s should effect the $y$s as much as past $X$s (and the order of the $X$s would be important to the prediction) so I decided I would try a bidirectional LSTM because that seemed like it made sense. I made the most basic kind possible in keras:

from keras.models import Sequential
from keras.layers import LSTM, Bidirectional, Dense

model = Sequential()

model.add(Bidirectional(LSTM(units=50, return_sequences=True), 
                        input_shape=(max_sequence_length, dim_inputs)))

model.add(Dense(dim_outputs, activation='softmax'))

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

I encoded the $X$s and $y$s as one hot vectors, padded the sequences to all the same length and set it running.

The accuracy was stupidly low so I decided to see if I had made a mistake in the definition of the model and assumed that a good thing to do would be to give it very predictable sequences and see whether it was able to predict them. It managed the identity but it couldn't do basic stuff like mapping $X$s to a constant vector (as it trained the accuracy would go down rapidly, so it did better at the start when it was randomised). I tried fooling around with stuff like the learning rate or the number of hidden units but the basic behaviour (getting worse with training) persisted.

I assume it is a basic problem with the definition of a model but I've also seen some study on DNNs having a hard time predicting basic linear functions like the identity if they contain non-linear activation functions (like softmax, which I put in the dense layer).

In case it's relevant, there are $20$ categories in for the inputs and $10$ for the outputs.

Is that what's happening here? Or the definition of my LSTM is just really far off the mark? Thank you!



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.