I'm using a lstm and feed-forward network to classify text.

I convert the text into one-hot vectors and feed each into the lstm so I can summarise it as a single representation. Then I feed it to the other network.

But how do I train the lstm? I just want to sequence classify the text— should I feed it without training? I just want to represent the passage as a single item I can feed into the input layer of the classifier.

I would greatly appreciate any advice with this!


So I have an lstm and a classifier. I take all the outputs of the lstm and mean-pool them, then I feed that average into the classifier.

My issue is that I don't know how to train the lstm or the classifier. I know what the input should be for the lstm and what the output of the classifier should be for that input. Since they are two separate networks that are just being activated sequentially, I need to know and don't know what the ideal-output should be for the lstm, which would also be the input for the classifier. Is there a way to do this?


1 Answer 1


The best place to start with LSTMs is the blog post of A. Karpathy http://karpathy.github.io/2015/05/21/rnn-effectiveness/. If you are using Torch7 (which I would strongly suggest) the source code is available at github https://github.com/karpathy/char-rnn.

I would also try to alter your model a bit. I would use a many-to-one approach so that you input words through a lookup table and add a special word at the end of each sequence, so that only when you input the "end of the sequence" sign you will read the classification output and calculate the error based on your training criterion. This way you would train directly under a supervised context.

On the other hand, a simpler approach would be to use paragraph2vec (https://radimrehurek.com/gensim/models/doc2vec.html) to extract features for your input text and then run a classifier on top of your features. Paragraph vector feature extraction is very simple and in python it would be:

class LabeledLineSentence(object):
    def __init__(self, filename):
        self.filename = filename

    def __iter__(self):
        for uid, line in enumerate(open(self.filename)):
            yield LabeledSentence(words=line.split(), labels=['TXT_%s' % uid])

sentences = LabeledLineSentence('your_text.txt')

model = Doc2Vec(alpha=0.025, min_alpha=0.025, size=50, window=5, min_count=5, dm=1, workers=8, sample=1e-5)

for epoch in range(epochs):
    except (KeyboardInterrupt, SystemExit):
  • $\begingroup$ Thank you for replying. I will consider those. Do you have any advice towards the specific issue in my question— I have updated it. $\endgroup$
    – wordSmith
    Jul 5, 2015 at 11:48
  • $\begingroup$ I don't think that your described procedure would produce any results. In what will you train against the LSTM? I'm not sure I understand why would use an LSTM in this case for unsupervised feature learning for a whole sentence. Do you have any relevant literature on your approach I could help you with? This might be of your interest as well arxiv.org/abs/1306.3584. $\endgroup$ Jul 5, 2015 at 12:31
  • $\begingroup$ I will train the lstm based on a data set of past passages of text and their classes. I'm not intending to use unsupervised learning. I want to manually train it, but don't know how. Here is my implementation of the lstm and classifier without the machine-learning library, which I know works: pastebin.com/63Cqrnef the lstm has a function of deepActivate that activate the lstm and then the classifier as I mentioned in my quesiton. Here's something like what I am trying to implement: deeplearning.net/tutorial/lstm.html $\endgroup$
    – wordSmith
    Jul 5, 2015 at 12:42
  • $\begingroup$ but when I tried to activate them both as one network, I got undefined from each of the output layers. More on that is here: stats.stackexchange.com/q/159922/81435 $\endgroup$
    – wordSmith
    Jul 5, 2015 at 12:44
  • 1
    $\begingroup$ Thank you very much! You provided much more help than was required. Thanks for going above and beyond. $\endgroup$
    – wordSmith
    Jul 5, 2015 at 17:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.