Combining two sequences for text classification I'm doing text classification on comments posted on articles/stories. The two human-labeled classes are appropriate and not appropriate (not the same as happy/angry or any "sentiment" specific mapping). A message/comment can be inappropriate if it's rude, aggressive, etc., but also if it's off-topic (ex: the article is about sports, but the comment is about politics).
What I'm wanting to do is input not only the comment into the model, but also the story associated with it, so the model can learn if a comment is off-topic.
How would I go about incorporating a 200 word comment AND a 1,000 word story into the same model?
I'm currently experimenting with the following architectures that takes the comment as input, and outputs two units (two class classification), based on papers on the subject from the last 3-5 years:
# One or more lstm layers
[Text sequence as words] => [embedding] => [bi-dir lstm] => [FC] => [sortmax]

CNN to lstm
[Text sequence as words] => [embedding] => [conv => maxpool] => [bi-dir lstm] => [FC] => [sortmax]

I'm thinking of inputting both sequences through the same embedding, but from there, not sure how to combine them - use separate conv blocks and combine the output of the lstm or something else..
 A: You have many options and it depends on many things (dataset size, compute you have access to, etc.).
First, you can choose an architecture that takes as inputs a sequence of words and that outputs a representation of a sequence. For example, a bidirectional-LSTM followed by max-pooling would do, or a CNN followed by max-pooling. (Not both sequentially as you wrote, though. I think you're a bit confused by what the CNN would output.) I would try with a pre-trained model in order to leverage huge corpora without supervision and without having to train too much myself. For example, BERT uses a special [CLS] token that is supposed to be a global sentence/document vector (trained to predict the next sentence), but there are other, more modern variants.
Then, you can encode both the document and the comment separately, concatenate the representation and train a classifier (logistic regression or MLP) on top.
Another option is to concatenate the two documents and use the [CLS] vector (or the equivalent in whatever Transformer-based model you use) to do your binary classification.
In both case, an orthogonal choice is whether to fine-tune your Transformer/CNN/BiLSTM model or not.
