I'm training a FCN on 550K datapoints (90/10 train-test split) and tracking training error, testing error, and actual MAE (un-z-scored true error project cares about) over each epoch. Below is plots for each:
The testing error + MAE jump from decent to incredibly off at around Epoch 27. Training loss seems to be generally decreasing.
train_set = Dataset(train_data,train_labels) # Dataset class just holds data/labels together train_sampler = RandomSampler(train_set) args = Params(batch_size=861, epochs=200, lr=0.01, momentum=0.5, seed=1, cuda=True, log_interval=5) train_loader = DataLoader(train_set, batch_size=args.batch_size, shuffle=None, sampler=train_sampler)
Also using z-scoring for regularization, early stopping, mse_loss on training/testing loss. Model is 7 layer FCN:
class Net(nn.Module): def __init__(self): super(Net, self).__init__() self.fc1 = nn.Linear(52, 100) self.fc2 = nn.Linear(100, 100) self.fc3 = nn.Linear(100, 100) self.fc4 = nn.Linear(100, 100) self.fc5 = nn.Linear(100, 100) self.fc6 = nn.Linear(100, 50) self.fc7 = nn.Linear(50, 1) self.bn1 = nn.BatchNorm1d(100) self.bn2 = nn.BatchNorm1d(100) self.bn3 = nn.BatchNorm1d(100) self.bn4 = nn.BatchNorm1d(100) self.bn5 = nn.BatchNorm1d(100) self.bn6 = nn.BatchNorm1d(50) def forward(self, x): x = self.bn1(Func.relu(self.fc1(x))) x = self.bn2(Func.relu(self.fc2(x))) x = self.bn3(Func.relu(self.fc3(x))) x = self.bn4(Func.relu(self.fc4(x))) x = self.bn5(Func.relu(self.fc5(x))) x = self.bn6(Func.relu(self.fc6(x))) x = self.fc7(x) return x
First thought is overfitting but training loss doesn't seem to be too dramatic. In addition, output on test set is almost all (95%) the same value.
Any ideas what would cause such incredible spikes later in the training process?