I wanted to use CNNs to classify sentences. The sentences are varying length. I am going to use standard Word Embeddings (any sort of pre-trained vectors) as features for each word then concatenate everything to represent one sentence. My issue is that sentences might have different lengths which leads to different layers having different dimensions based on the # of word we have. Then that make some layer (usually my fully connected layer) screw up because it never knows what size vector its going to receive. I really do not want to pad my sentences with nonsense to fix the issue. Is there a way to solve this issue without padding sentences with irrelevant stuff so that the final fully connected classification layer works fine?

I found the following paper:


and in the model section they have a part which I suspect might be what I am looking for but I am unable to understand it (i.e. I feel it fixes some layer to have a certain size which is what I need for my FC layer to work and thus my classification layer to work).

We then apply a max-overtime pooling operation (Collobert et al., 2011) over the feature map and take the maximum value ^c = maxfcg as the feature corresponding to this particular filter. The idea is to capture the most important feature—one with the highest value—for each feature map. This pooling scheme naturally deals with variable sentence lengths.

in particular it says:

This pooling scheme naturally deals with variable sentence lengths.

  • $\begingroup$ cross-posted: quora.com/unanswered/… $\endgroup$ Commented Apr 25, 2019 at 22:32
  • $\begingroup$ I see you've added a bounty to this question requesting "reputable sources." Bolton Bailey's answer is the correct one, especially in the context of the article you've cited in your question. Can you explain how the cited source is not "reputable"? What would a "reputable" source be in response to your question? $\endgroup$
    – Sycorax
    Commented Oct 9, 2020 at 3:32

1 Answer 1


Quoting from the reference (Collobert et al., 2011) from the paper you mentioned

The size of the output (6) depends on the number of words in the sentence fed to the network. Local feature vectors extracted by the convolutional layers have to be combined to obtain a global feature vector, with a fixed size independent of the sentence length, in order to apply subsequent standard affine layers. Traditional convolutional networks often apply an average (possibly weighted) or a max operation over the “time” t of the sequence (6). (Here, “time” just means the position in the sentence, this term stems from the use of convolutional layers in, for example, speech data where the sequence occurs over time.) The average operation does not make much sense in our case, as in general most words in the sentence do not have any influence on the semantic role of a given word to tag. Instead, we used a max approach, which forces the network to capture the most useful local features produced by the convolutional layers

So the solution being suggested here is for the convolutional network to produce a matrix of dimensionality (number of words in sentence) x (number of features in the last convolution), and then to take the maximum over rows of this matrix to get a single vector of features for the whole sentence of fixed dimension.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.