Deep Learning - why are we doing the forward pass on the whole dataset when using SGD

Question

When reading the torch.optim documentation of PyTorch (https://pytorch.org/docs/master/optim.html), they do the forward pass on the whole dataset when using SGD but this Cross Validated answer clearly states that we shouldn't (https://stats.stackexchange.com/a/331271/353716).

PyTorch documentation:

optim.SGD([{'params': model.base.parameters()}, {'params': model.classifier.parameters(), 'lr': 1e-3}], lr=1e-2, momentum=0.9)
for input, target in dataset:
    optimizer.zero_grad()
    output = model(input)
    loss = loss_fn(output, target)
    loss.backward()
    optimizer.step()

And here (Minimal working example of optim.SGD (https://discuss.pytorch.org/t/minimal-working-example-of-optim-sgd/11623)), they do the same:

# Data
X = Variable(torch.randn(N, 1))

# (noisy) Target values that we want to learn.
t = A * X + b + Variable(torch.randn(N, 1) * error)

# Creating a model, making the optimizer, defining loss
model = nn.Linear(1, 1)
optimizer = optim.SGD(model.parameters(), lr=0.05)
loss_fn = nn.MSELoss()

# Run training
niter = 50
for _ in range(0, niter):
    optimizer.zero_grad()
    predictions = model(X)
    loss = loss_fn(predictions, t)
    loss.backward()
    optimizer.step()

    print("-" * 50)
    print("error = {}".format(loss.data[0]))
    print("learned A = {}".format(list(model.parameters())[0].data[0, 0]))
    print("learned b = {}".format(list(model.parameters())[1].data[0]))

But if I understood SGD well, SGD only computes the gradient on a subset of the whole training dataset and then updates the parameters based on this gradient. Why are using the whole dataset in the forward pass then? (Basically what the Cross Validated question I was referencing is asking except it wasn't talking about PyTorch at all). If we are, we might as well do "normal" gradient descent? Otherwise we've computed stuff for no reason in the forward pass?

Answer 1

You understood it correctly: SGD should compute gradients on a subset of the data:

• The PyTorch documentation you refer to actually does use subsets. It pushes the stochasticity to the limit by computing gradients on single samples (for input, target in dataset).
• The answer in the forums that you brought up, probably just uses X as a dummy variable in order not to have to bother with loading any actual data. I wouldn't interpret this as an example of training on the entire dataset.

This being said, I do agree that PyTorch naming is a bit confusing in this regard. The implementation of SGD is actually just gradient descent. The stochasticity comes purely from using (shuffled) mini-batches for each optimizer step. Without this dataloading aspect, the optimisation algorithm would effectively perform GD, rather than SGD.