Why shouldn't you mix variable size inputs in the same minibatch? I am trying to build a CNN-LSTM architecure in tf.keras that classifies sequences of varying sizes. My training data is highly variable and I would have to crop/pad sequences in order to create similar sized minibatches of the same sequence length respectively.
Hence my question:
How important is it to have minibatches of sequences of the same length? Is this due to some inherent property of the way data the tensors are processed (i.e. will I get an error) or is it more a performance consideration (i.e. parallelizability) that can be discarded if needs be?
Many thanks!
Tom
 A: This is definitely about performance. Without batching by length, you perform wasteful computation whose results you ignore anyway.
Remember that a tensor has to be rectangular—you can't have a ragged edge. (Generalize this to higher dimensions as needed.) But sequences are different lengths. Padding is used to remove the ragged edge, but computation on the padding is useless. We want to minimize wasted computation, which is why we group by length.

Consider the following dataset of six sequences. Each row is one sequence, and each X represents one item in the sequence:
X
XXXXX
XX
XX
XXX
X

Remember that our CNN runs along the time axis of our sequence. For batched sequences, we run to the length of our batch's longest sequence. When sequences are different lengths, we use padding. But batching the data differently can lead to different total lengths of batches.
Base amount of work (no padding, no batching)
Let's look at the work that needs to be done. We have to process 14 items across our 6 sequences. By batching, we can speed this up—but we might end up doing extra work.
Choice 1: No grouping by length
Let's just batch as-is, with a batch size of 2. I'll use . to represent padding.
X....
XXXXX

XX
XX

XXX
X..

If we run our CNN on this, we wind up doing 5+2+3=10 units of work walking along these sequences. That might not be optimal; look at all that unnecessary work we did on the first sequence!
Choice 2: Grouping by length
Let's group by length as best as possible, keeping the batch size 2.
X
X

XX
XX

XXXXX
XXX..

This time, we only do 1+2+5=8 units of work. The other option was 25% slower, wasting its time on computation we never needed to do. It ran on the padding, so we'd wind up throwing it away anyway.

This was obviously a toy example. For larger datasets the problem is more extreme—because the ratio between the longest and average sequence lengths in a batch can be larger.
