I'm training a FCN on 550K datapoints (90/10 train-test split) and tracking training error, testing error, and actual MAE (un-z-scored true error project cares about) over each epoch. Below is plots for each:

enter image description here

The testing error + MAE jump from decent to incredibly off at around Epoch 27. Training loss seems to be generally decreasing.

Parameters used:

    train_set = Dataset(train_data,train_labels) # Dataset class just holds data/labels together
    train_sampler = RandomSampler(train_set)

    args = Params(batch_size=861, epochs=200, lr=0.01, momentum=0.5, seed=1, cuda=True, log_interval=5)
    train_loader = DataLoader(train_set, batch_size=args.batch_size, shuffle=None, sampler=train_sampler)

Also using z-scoring for regularization, early stopping, mse_loss on training/testing loss. Model is 7 layer FCN:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(52, 100)
        self.fc2 = nn.Linear(100, 100)
        self.fc3 = nn.Linear(100, 100)
        self.fc4 = nn.Linear(100, 100)
        self.fc5 = nn.Linear(100, 100)
        self.fc6 = nn.Linear(100, 50)
        self.fc7 = nn.Linear(50, 1)

        self.bn1 = nn.BatchNorm1d(100)
        self.bn2 = nn.BatchNorm1d(100)
        self.bn3 = nn.BatchNorm1d(100)
        self.bn4 = nn.BatchNorm1d(100)
        self.bn5 = nn.BatchNorm1d(100)
        self.bn6 = nn.BatchNorm1d(50)

    def forward(self, x):
        x = self.bn1(Func.relu(self.fc1(x)))
        x = self.bn2(Func.relu(self.fc2(x)))
        x = self.bn3(Func.relu(self.fc3(x)))
        x = self.bn4(Func.relu(self.fc4(x)))
        x = self.bn5(Func.relu(self.fc5(x)))
        x = self.bn6(Func.relu(self.fc6(x)))
        x = self.fc7(x)
        return x

First thought is overfitting but training loss doesn't seem to be too dramatic. In addition, output on test set is almost all (95%) the same value.

Any ideas what would cause such incredible spikes later in the training process?

  • $\begingroup$ What definition of overfitting are you using? Do your observations of wildly increasing holdout loss fit that definition? Why or why not? (As an aside, you may wish to learn more about the nn.Sequential class in pytorch.) $\endgroup$
    – Sycorax
    Feb 26, 2020 at 21:06
  • $\begingroup$ @SycoraxsaysReinstateMonica by overfitting I simply mean that the model memorizes the training set to a point that testing loss suffers. I don't see that happening because typically you would see training loss start to taper off and testing loss gradually/rapidly increase simultaneously. In this case is seems to single epochs of huge test loss late in training. Thanks for referencing nn.Sequential! Have used it previously $\endgroup$
    – Adam
    Feb 26, 2020 at 21:33
  • $\begingroup$ I didn't notice before that the scale of the test axis is measured in $10^{11}$. This suggests to me that for some reason there's a division-by-0 somewhere, possibly because the batch norm denominator has vanished. It's possible to get a unit with 0 variance: it suggests a dead relu for the test data but not the train data. $\endgroup$
    – Sycorax
    Feb 26, 2020 at 21:37

1 Answer 1


It makes sense that once 27 epochs have got completed some features are extracted which suddenly generalizes the model better which make the test loss suddenly to spike up.

  • $\begingroup$ Thanks for the response! I would understand if test loss spiked some amount as new features are extracted-- even an order of magnitude would be fine in some cases. But the loss is spiking from near zero the epoch before ,~0.002, to 1.6e11, and back down to near 0. That's a spike 10 trillion times larger as the previous epic. It's inconceivable that a spike of that magnitude is due to feature extraction at epoch 27 $\endgroup$
    – Adam
    Mar 6, 2020 at 19:17
  • $\begingroup$ I do understand that,but this is the one kind of reasoning which is more convincing as neural network is a black box. $\endgroup$ Mar 12, 2020 at 5:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.