1
$\begingroup$

I'm setting up a CNN that can handle differently-sized examples so long as all examples within the same batch are of the same size.

As a trade-off for this flexibility, I need to fix the batch size. (Only one dimension of the tensor can be dynamic).

In the current implementation, the training examples are split into length-keyed buckets. One iteration then consists of randomly sampling BATCH_SIZE many examples from each bucket and feeding these batches - in turn - through the network.

I am unsure, however, what I should do if I don't have enough samples of a certain length to fill that batch with distinct examples. Say, I may have 55'000 examples but only 7 of length 2 when the batch size is 50.

Should I drop lengths for which I don't have "enough" examples?

Should I just keep sampling from however many examples I have until I have a full batch (obviously repeating some examples)?

Pre-process the input examples and choose smallest possible batch size (at risk of that turning out to be 1)?

Or something else entirely?

I'm currently leaning towards "just keep sampling until the batch is full" because we're losing information otherwise.

However, I am also worried that this might skew the network somehow.

Somehow, I say, because I quite simply don't know and am ill-equipped to make an educated guess.

Are these concerns founded?

$\endgroup$
0
$\begingroup$

Without knowing what kind of data you have, I would suggest that you try zero padding your input data.

In this way, all your examples have the same size and your network is more robust to the different sizes of inputs, even to those that you do not have any training examples for.

In that case, you would fix your input size to the largest training example you have (or a larger number if you know that your data could come in a larger format). With this, you would be able to vary your batch size.

$\endgroup$
  • $\begingroup$ It's a classifier, classifying differently-sized vectors of integers. And your suggestion kind of defeats the purpose of having a network capable of taking differently-sized inputs. $\endgroup$ – User1291 May 15 '17 at 11:24
  • $\begingroup$ and those integers are raw readings? Or are they processed from somewhere? $\endgroup$ – BrunoGL May 15 '17 at 11:25
  • $\begingroup$ Tokenised (English) sentences (each word assigned its own number). $\endgroup$ – User1291 May 15 '17 at 11:26
  • $\begingroup$ Then I would suggest that you do not map any word to 0. Start mapping from 1. Then see the largest size of your input (for example 20). Then all your input could have that size. If your input has length 2, then the first two integers are the two words and the other 18 are 0. I know it seems like you are limiting variability of your input but this works. Then only limiting factor is the maximum size of your input. $\endgroup$ – BrunoGL May 15 '17 at 11:31
  • 1
    $\begingroup$ Common methods like tf-idf use sparse inputs with 0's and it is one of the best features for text classification around. Hint: use tf-idf instead of your integers. $\endgroup$ – BrunoGL May 15 '17 at 11:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.