I'm setting up a CNN that can handle differently-sized examples so long as all examples within the same batch are of the same size.
As a trade-off for this flexibility, I need to fix the batch size. (Only one dimension of the tensor can be dynamic).
In the current implementation, the training examples are split into length-keyed buckets. One iteration then consists of randomly sampling
BATCH_SIZE many examples from each bucket and feeding these batches - in turn - through the network.
I am unsure, however, what I should do if I don't have enough samples of a certain length to fill that batch with distinct examples. Say, I may have 55'000 examples but only 7 of length 2 when the batch size is 50.
Should I drop lengths for which I don't have "enough" examples?
Should I just keep sampling from however many examples I have until I have a full batch (obviously repeating some examples)?
Pre-process the input examples and choose smallest possible batch size (at risk of that turning out to be 1)?
Or something else entirely?
I'm currently leaning towards "just keep sampling until the batch is full" because we're losing information otherwise.
However, I am also worried that this might skew the network somehow.
Somehow, I say, because I quite simply don't know and am ill-equipped to make an educated guess.
Are these concerns founded?