I have minibatch gradient descent code in Tensorflow for function approximation, but I am unsure when to calculate the loss. First, I create batches for x and y data. Then, I shuffle both these batches every epoch. I work with 2 lists, train_batch_loss is used to store train losses for each batch, so I can then take the average over all batches and append it to the train_loss list. My code looks like this

train_loss = []
for epoch in range(number_of_epochs):
    x_train_batches, y_train_batches = createBatches(batch_size, Xdata, Ydata)
    number_of_batches = len(x_train_batches)
    x_train_batches = shuffle(x_train_batches)
    y_train_batches = shuffle(y_train_batches)
    train_batch_loss = []
    for batch in range(number_of_batches):
        t = sess.run([training], feed_dict = {X:x_train_batches[batch],Y:y_train_batches[batch]})
        train_batch_loss.append(sess.run(cost_function, feed_dict = {X:x_train_batches[batch],Y:y_train_batches[batch]}))

Is this the correct way to calculate losses for minibatch GD?


X and y data should be shuffled accordingly, so that the pairings are consistent in the minibatches (not evident in your code due the 2 separate shuffle calls). You might also want to compute the loss in an independent validation set (ie, never gets mixed with train batches across all epochs). I guess Xdata, Ydata in your code is only train data, right? Otherwise validation (and even worse, test) data would get totally mixed-up

  • $\begingroup$ Oh I didn't think about the shuffling problem, thanks! Yes, it is only train data, I didn't include validation set, for the sake of easier readability. I am mainly interested if the main idea of loss computation is correct, meaning I compute losses for each batch and then take the average? $\endgroup$ – user430953 Feb 17 '19 at 11:42
  • $\begingroup$ That would be fine yes. $\endgroup$ – Tom Feb 17 '19 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.