0
$\begingroup$

I am just starting in machine learning and wanted to make sure I understand the relationship between batch size and the number of features in the input data. I am referring specifically to the last chunk of code on this getting started guide provided by Keras:

https://keras.io/getting-started/sequential-model-guide

and the FAQ on stateful LSTM's:

https://keras.io/getting-started/faq/#how-can-i-use-stateful-rnns

where it said:

explicitly specify the batch size you are using, by passing a batch_size argument to the first layer in your model. E.g. batch_size=32 for a 32-samples batch of sequences of 10 timesteps with 16 features per timestep.

Now, both in the getting started guide and the FAQ, the batch size is twice the size of the feature set of the input data... I just want to confirm that this doubling is only used as a "for instance" and the batch size could have been three times or four times the feature set (or data dimension), but that the batch size must be a multiple of the feature set and not something where there were 5 features and you could choose to go with a batch size of 7 or 12, right?

from keras.models import Sequential
from keras.layers import LSTM, Dense
import numpy as np

data_dim = 16
timesteps = 8
num_classes = 10
batch_size = 32

# Expected input batch shape: (batch_size, timesteps, data_dim)
# Note that we have to provide the full batch_input_shape since the network is stateful.
# the sample of index i in batch k is the follow-up for the sample i in batch k-1.
model = Sequential()
model.add(LSTM(32, return_sequences=True, stateful=True,
               batch_input_shape=(batch_size, timesteps, data_dim)))
model.add(LSTM(32, return_sequences=True, stateful=True))
model.add(LSTM(32, stateful=True))
model.add(Dense(10, activation='softmax'))

model.compile(loss='categorical_crossentropy',
              optimizer='rmsprop',
              metrics=['accuracy'])

# Generate dummy training data
x_train = np.random.random((batch_size * 10, timesteps, data_dim))
y_train = np.random.random((batch_size * 10, num_classes))

# Generate dummy validation data
x_val = np.random.random((batch_size * 3, timesteps, data_dim))
y_val = np.random.random((batch_size * 3, num_classes))

model.fit(x_train, y_train,
          batch_size=batch_size, epochs=5, shuffle=False,
          validation_data=(x_val, y_val))

Also, that the number of units initialized in the LSTM layers only just so happens to be 32, the same size as the batch size and that this number really could be anything, such as 40 or 50, etc.

$\endgroup$
1
$\begingroup$

The batch size and number of units in a layer can be arbitrary and don't have to be multiples of each other. You will usually see powers of two just because they have nice properties for GPUs.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.