Sentence classification using an RNN, without classifying each sentence separately

Question

Suppose you are trying to do sentence classification. That is, given a block of text with many sentences, I want to output the "class" of each sentence in order. For example, suppose there are classes A, B, and C, and this is the input text:

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Pellentesque eu tincidunt tortor aliquam nulla. Fermentum leo vel orci porta non pulvinar neque laoreet. Proin libero nunc consequat interdum varius.

A potential output would be: B C A C. Since there are 4 sentences, the output would be 4 classes, one for each sentence.

A quick Google search for sentence classification using an RNN gives many results (e.g. here). As far as I can tell, they propose two steps: first use an RNN to embed each sentence into some space, then have a feed-forward neural network which classifies each embedded sentence into one of the three classes.

This does not work for me, because it treats each sentence separately. That is, to classify a given sentence, I want to use information from the previous sentences. And not just the predicted "class" of the previous sentences, but rather their actual text.

Ideally, I would have a character-based RNN that takes the input, character by character, and upon reaching the end of a sentence (i.e. the period), it outputs one of the four classes. The appeal of this approach is that the RNN would "remember" the previous sentences as well, and that could help with classification of a given sentence.

This is, in some sense, a "sequence to sequence" RNN. It takes the sequence of sentences and outputs a sequence of corresponding classes.

Does this sort of architecture exist?

Answer 1

Add another RNN.

Your first RNN outputs a set of features for each sentence, based on words or characters. (I used a CNN myself.)

Your second (bi-directional) RNN takes those features and outputs features based on the features of the current and other sentences.

Your fully-connected classifier then assigns a classification to each line. You’ll need to make sure the classifier is getting input at each time step (i.e. for each sentence) and not just the usual default of results for the last sentence (i.e. for all of your sentences at once).

I guess you could also use seq2seq techniques that involve transformers and attention, but your task does have a natural input and output cadence, so in that way isn’t like translating English to French. You have the same number of input sentences as output classifications, and they will be made in the same order.