Why is Testing Error Spiking late in the training process?

Question

I'm training a FCN on 550K datapoints (90/10 train-test split) and tracking training error, testing error, and actual MAE (un-z-scored true error project cares about) over each epoch. Below is plots for each:

The testing error + MAE jump from decent to incredibly off at around Epoch 27. Training loss seems to be generally decreasing.

Parameters used:

    train_set = Dataset(train_data,train_labels) # Dataset class just holds data/labels together
    train_sampler = RandomSampler(train_set)

    args = Params(batch_size=861, epochs=200, lr=0.01, momentum=0.5, seed=1, cuda=True, log_interval=5)
    train_loader = DataLoader(train_set, batch_size=args.batch_size, shuffle=None, sampler=train_sampler)

Also using z-scoring for regularization, early stopping, mse_loss on training/testing loss. Model is 7 layer FCN:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(52, 100)
        self.fc2 = nn.Linear(100, 100)
        self.fc3 = nn.Linear(100, 100)
        self.fc4 = nn.Linear(100, 100)
        self.fc5 = nn.Linear(100, 100)
        self.fc6 = nn.Linear(100, 50)
        self.fc7 = nn.Linear(50, 1)

        self.bn1 = nn.BatchNorm1d(100)
        self.bn2 = nn.BatchNorm1d(100)
        self.bn3 = nn.BatchNorm1d(100)
        self.bn4 = nn.BatchNorm1d(100)
        self.bn5 = nn.BatchNorm1d(100)
        self.bn6 = nn.BatchNorm1d(50)

    def forward(self, x):
        x = self.bn1(Func.relu(self.fc1(x)))
        x = self.bn2(Func.relu(self.fc2(x)))
        x = self.bn3(Func.relu(self.fc3(x)))
        x = self.bn4(Func.relu(self.fc4(x)))
        x = self.bn5(Func.relu(self.fc5(x)))
        x = self.bn6(Func.relu(self.fc6(x)))
        x = self.fc7(x)
        return x

First thought is overfitting but training loss doesn't seem to be too dramatic. In addition, output on test set is almost all (95%) the same value.

Any ideas what would cause such incredible spikes later in the training process?

Answer 1

It makes sense that once 27 epochs have got completed some features are extracted which suddenly generalizes the model better which make the test loss suddenly to spike up.