Let me put my question first. For a time-series prediciton, is it possible to overfit even within the first epoch, when training data and validation data should all "new" to model?
Features and Model
I'm building model to perform time series predicion for sales data. The data is comming from one of the Kaggle competition Rosseman Sales. The data provide each day's sales number and its related OPEN and PROMOTION status basically. The features I put in are (assume T-0D is the date predict):
Input:
- SalesDataPart1: T-30D ~ T-1D sales data
- SalesDataPart2: T-370D ~ T-350D historical sales data (to catch the yearly pattern)
- OpenStatus: T-6D ~ T-1D store open status
- PromoStatus: T-6D ~ T-1D store promotion status
- CurrentOpenStatus: T-0D store open status
Output:
- T-0D sales prediction
Inside the model, at the first stage, I put 3 LSTM. All SalesData feed into one LSTM, OpenStatus and Promo Status feed into the reset two LSTM. All three LSTM output feed into one full connected layer and then generate a singal output activate with a sigmoid function. Finally this sigmoid output is mutiply with the predicted day OpenStatus, and generate the final output. Detail as below:
Overall speaking, input features is around 70. All LSTM are single layer, single direction, with a hidden state around half of its input. the full connected network is of 32 hidden node. Drop out is apply between models, with a rate of 0.5.
The data provided by kaggle enable me to generate around 550 time-series training sets configure as above. I further divide training and validation data sets with a ratio of 0.3, randomly selected. DataLoader is also constructed with batchsize of 1, shuffled.
Training
During training, one random entry is feed to model for each step, and validation performed every 20 steps. Validation will run through all the validation datasets. The hidden states is not preserve during steps.
And here is the problem. Even though I only have one Epoch, the training loss will drop along with steps. However, the validation loss will continue to raise. I understand that the network might "remember" the training sets so overfitting happen, but how can it happen within a single epoch, when model haven't train with all the data? At least I'll expect the validation loss is as low as the training loss within single epoch, as they are all "new" to the model.
Plot for Training/Validation Loss
Note that the x-axis is steps insted of epochs. Each epoch have around 550 steps. So you can observe that during the first epoch, training loss is constantly decresing while the validation loss increase.
Below are the code to train the model..
# criterion = nn.BCELoss()
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
model.train()
for i in range(epochs):
h = model.init_hidden()
for lookback, attention, openSta, currentopen, promo, labels in trainloader:
steps += 1
# h = tuple([e.data for e in h])
h, h1, h2 = model.init_hidden()
# Move data to default device
lookback = lookback.to(device)
attention = attention.to(device)
openSta = openSta.to(device)
currentopen = currentopen.to(device)
promo = promo.to(device)
labels = labels.to(device)
optimizer.zero_grad()
output, h, h1, h2 = model.forward(lookback, attention, openSta, currentopen, promo, h, h1, h2)
loss = criterion(output, labels)
loss.backward()
optimizer.step()
running_loss += loss.item()
# record the last printPrediction prediction and label for printing
if steps > len(trainloader) - printPrediction:
printPredictData.append(output.to(torch.device("cpu")).detach().numpy()[0][0])
printLabelData.append(labels.to(torch.device("cpu")).numpy()[0][0])
if steps % print_every == 0:
test_loss = 0
accuracy= 0
model.eval()
with torch.no_grad():
for lookback, attention, openSta, currentopen, promo, labels in validloader:
# Move data to default device
lookback = lookback.to(device)
attention = attention.to(device)
openSta = openSta.to(device)
currentopen = openSta.to(device)
promo = promo.to(device)
labels = openSta.to(device)
h, h1, h2 = model.init_hidden()
output, h_val, h1_val, h2_val = model.forward(lookback, attention, openSta, currentopen, promo, h, h1, h2)
batch_loss = criterion(output, labels)
test_loss += batch_loss.item()
trainingLoss.append(running_loss/print_every)
validLoss.append(test_loss/len(validloader))
print("Epoch: {}/{}...".format(i+1, epochs),
"Step: {}...".format(steps),
"Train Loss: {:.6f}...".format(running_loss/print_every),
"Step Loss: {:.6f}...".format(loss.item()),
"Val Loss: {:.6f}".format(test_loss/len(validloader)))
running_loss = 0
model.train()
