1
$\begingroup$

I am trying to understand the best practice for handling different lengths of sequences in NLP tasks. Lets consider an example of convolution on sequences followed by max pool layer.

We can handle this in two ways.

  1. Make sure each minibatch contains sequences of same length - in this case we don't need any masking after convolution before applying max pooling.

  2. Each minibatch can contain sequences of different length - in this case we need to apply masking (to padding added to the each sequence to make it constant length) after convolution before applying max pooling.

Option 1 is simple to implement because it is handled only at data side, option 2 is more complicated because masking should be done correctly.

But I feel option 1 does not feed randomized data to the network. Especially in cases where modelling long sequences is more harder than short sequences, in these scenario, loss of long sequence minibatch will be higher than that of short sequence minibatch. Will this lead to any convergence issues ?

Can anyone share the best practices in deciding ?

Improve this question
$\endgroup$
2
  • $\begingroup$ I agree that (1) doesn't sound like a good option. A possible, equivalent alternative to (2) would be to individually pass each sequence in the minibatch through the convolution+max pooling layers, then aggregate the results into a matrix for processing by further layers. How this compares in ease/efficiency depends on your implementation. $\endgroup$ – user20160 Dec 20 '18 at 22:45
  • $\begingroup$ I think your solution will be very inefficient especially when using GPU, because we are not allowing parallelism in computation. What is the most common way of handling ? I expect this problem to be more common but i don't see lot of article talking about this. $\endgroup$ – Kumaran Dec 22 '18 at 20:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.