# Loss during minibatch gradient descent

I have minibatch gradient descent code in Tensorflow for function approximation, but I am unsure when to calculate the loss. First, I create batches for x and y data. Then, I shuffle both these batches every epoch. I work with 2 lists, train_batch_loss is used to store train losses for each batch, so I can then take the average over all batches and append it to the train_loss list. My code looks like this

train_loss = []
for epoch in range(number_of_epochs):
x_train_batches, y_train_batches = createBatches(batch_size, Xdata, Ydata)
number_of_batches = len(x_train_batches)
x_train_batches = shuffle(x_train_batches)
y_train_batches = shuffle(y_train_batches)
train_batch_loss = []
for batch in range(number_of_batches):
t = sess.run([training], feed_dict = {X:x_train_batches[batch],Y:y_train_batches[batch]})
train_batch_loss.append(sess.run(cost_function, feed_dict = {X:x_train_batches[batch],Y:y_train_batches[batch]}))
train_loss.append(sum(train_batch_loss)/len(train_batch_loss))


Is this the correct way to calculate losses for minibatch GD?

## 1 Answer

X and y data should be shuffled accordingly, so that the pairings are consistent in the minibatches (not evident in your code due the 2 separate shuffle calls). You might also want to compute the loss in an independent validation set (ie, never gets mixed with train batches across all epochs). I guess Xdata, Ydata in your code is only train data, right? Otherwise validation (and even worse, test) data would get totally mixed-up

• Oh I didn't think about the shuffling problem, thanks! Yes, it is only train data, I didn't include validation set, for the sake of easier readability. I am mainly interested if the main idea of loss computation is correct, meaning I compute losses for each batch and then take the average? – Crash Feb 17 '19 at 11:42
• That would be fine yes. – Tom Feb 17 '19 at 14:58