2
$\begingroup$

When reading the torch.optim documentation of PyTorch (https://pytorch.org/docs/master/optim.html), they do the forward pass on the whole dataset when using SGD but this Cross Validated answer clearly states that we shouldn't (https://stats.stackexchange.com/a/331271/353716).

PyTorch documentation:

optim.SGD([{'params': model.base.parameters()}, {'params': model.classifier.parameters(), 'lr': 1e-3}], lr=1e-2, momentum=0.9)
for input, target in dataset:
    optimizer.zero_grad()
    output = model(input)
    loss = loss_fn(output, target)
    loss.backward()
    optimizer.step()

And here (Minimal working example of optim.SGD (https://discuss.pytorch.org/t/minimal-working-example-of-optim-sgd/11623)), they do the same:

# Data
X = Variable(torch.randn(N, 1))

# (noisy) Target values that we want to learn.
t = A * X + b + Variable(torch.randn(N, 1) * error)

# Creating a model, making the optimizer, defining loss
model = nn.Linear(1, 1)
optimizer = optim.SGD(model.parameters(), lr=0.05)
loss_fn = nn.MSELoss()

# Run training
niter = 50
for _ in range(0, niter):
    optimizer.zero_grad()
    predictions = model(X)
    loss = loss_fn(predictions, t)
    loss.backward()
    optimizer.step()

    print("-" * 50)
    print("error = {}".format(loss.data[0]))
    print("learned A = {}".format(list(model.parameters())[0].data[0, 0]))
    print("learned b = {}".format(list(model.parameters())[1].data[0]))

But if I understood SGD well, SGD only computes the gradient on a subset of the whole training dataset and then updates the parameters based on this gradient. Why are using the whole dataset in the forward pass then? (Basically what the Cross Validated question I was referencing is asking except it wasn't talking about PyTorch at all). If we are, we might as well do "normal" gradient descent? Otherwise we've computed stuff for no reason in the forward pass?

$\endgroup$

1 Answer 1

2
$\begingroup$

You understood it correctly: SGD should compute gradients on a subset of the data:

  • The PyTorch documentation you refer to actually does use subsets. It pushes the stochasticity to the limit by computing gradients on single samples (for input, target in dataset).
  • The answer in the forums that you brought up, probably just uses X as a dummy variable in order not to have to bother with loading any actual data. I wouldn't interpret this as an example of training on the entire dataset.

This being said, I do agree that PyTorch naming is a bit confusing in this regard. The implementation of SGD is actually just gradient descent. The stochasticity comes purely from using (shuffled) mini-batches for each optimizer step. Without this dataloading aspect, the optimisation algorithm would effectively perform GD, rather than SGD.

$\endgroup$
3
  • $\begingroup$ Aobut the PyTorch documentation: I read it wrong, I thought the for loop was for epochs, but I just misread it! Makes sense now. $\endgroup$ Jan 16, 2023 at 14:57
  • 1
    $\begingroup$ "The implementation of SGD is actually just gradient descent. The stochasticity comes purely from using (shuffled) mini-batches for each optimizer step. Without this dataloading aspect, the optimisation algorithm would effectively perform GD, rather than SGD." That's exactly what was bothering me! So much clearer now! $\endgroup$ Jan 16, 2023 at 14:57
  • $\begingroup$ That would be a way to do SGD with mini-batches? By doing train_loader = DataLoader(dataset=train_data, batch_size=16, shuffle=True) and then for inputs, targets in train_loader $\endgroup$ Jan 16, 2023 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.