2
$\begingroup$

Let me put my question first. For a time-series prediciton, is it possible to overfit even within the first epoch, when training data and validation data should all "new" to model?


Features and Model

I'm building model to perform time series predicion for sales data. The data is comming from one of the Kaggle competition Rosseman Sales. The data provide each day's sales number and its related OPEN and PROMOTION status basically. The features I put in are (assume T-0D is the date predict):

Input:

  • SalesDataPart1: T-30D ~ T-1D sales data
  • SalesDataPart2: T-370D ~ T-350D historical sales data (to catch the yearly pattern)
  • OpenStatus: T-6D ~ T-1D store open status
  • PromoStatus: T-6D ~ T-1D store promotion status
  • CurrentOpenStatus: T-0D store open status

Output:

  • T-0D sales prediction

Inside the model, at the first stage, I put 3 LSTM. All SalesData feed into one LSTM, OpenStatus and Promo Status feed into the reset two LSTM. All three LSTM output feed into one full connected layer and then generate a singal output activate with a sigmoid function. Finally this sigmoid output is mutiply with the predicted day OpenStatus, and generate the final output. Detail as below:

enter image description here

Overall speaking, input features is around 70. All LSTM are single layer, single direction, with a hidden state around half of its input. the full connected network is of 32 hidden node. Drop out is apply between models, with a rate of 0.5.

The data provided by kaggle enable me to generate around 550 time-series training sets configure as above. I further divide training and validation data sets with a ratio of 0.3, randomly selected. DataLoader is also constructed with batchsize of 1, shuffled.

Training

During training, one random entry is feed to model for each step, and validation performed every 20 steps. Validation will run through all the validation datasets. The hidden states is not preserve during steps.

And here is the problem. Even though I only have one Epoch, the training loss will drop along with steps. However, the validation loss will continue to raise. I understand that the network might "remember" the training sets so overfitting happen, but how can it happen within a single epoch, when model haven't train with all the data? At least I'll expect the validation loss is as low as the training loss within single epoch, as they are all "new" to the model.

Plot for Training/Validation Loss

Note that the x-axis is steps insted of epochs. Each epoch have around 550 steps. So you can observe that during the first epoch, training loss is constantly decresing while the validation loss increase.

enter image description here

Below are the code to train the model..

# criterion = nn.BCELoss()
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

model.train()
for i in range(epochs):
    h = model.init_hidden()

    for lookback, attention, openSta, currentopen, promo, labels in trainloader:
        steps += 1
        # h = tuple([e.data for e in h])
        h, h1, h2 = model.init_hidden()
        # Move data to default device
        lookback    = lookback.to(device)
        attention   = attention.to(device)
        openSta     = openSta.to(device)
        currentopen = currentopen.to(device)
        promo       = promo.to(device)
        labels      = labels.to(device)

        optimizer.zero_grad()

        output, h, h1, h2 = model.forward(lookback, attention, openSta, currentopen, promo, h, h1, h2)
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()

        # record the last printPrediction prediction and label for printing
        if steps > len(trainloader) - printPrediction:
            printPredictData.append(output.to(torch.device("cpu")).detach().numpy()[0][0])
            printLabelData.append(labels.to(torch.device("cpu")).numpy()[0][0])

        if steps % print_every == 0:
            test_loss = 0
            accuracy= 0
            model.eval()
            with torch.no_grad():
                for lookback, attention, openSta, currentopen, promo, labels in validloader:
                    # Move data to default device
                    lookback    = lookback.to(device)
                    attention   = attention.to(device)
                    openSta     = openSta.to(device)
                    currentopen = openSta.to(device)
                    promo       = promo.to(device)
                    labels      = openSta.to(device)

                    h, h1, h2 = model.init_hidden()
                    output, h_val, h1_val, h2_val = model.forward(lookback, attention, openSta, currentopen, promo, h, h1, h2)
                    batch_loss = criterion(output, labels)
                    test_loss += batch_loss.item()

            trainingLoss.append(running_loss/print_every)
            validLoss.append(test_loss/len(validloader))

            print("Epoch: {}/{}...".format(i+1, epochs),
                  "Step: {}...".format(steps),
                  "Train Loss: {:.6f}...".format(running_loss/print_every),
                  "Step Loss: {:.6f}...".format(loss.item()),
                  "Val Loss: {:.6f}".format(test_loss/len(validloader)))
            running_loss = 0
            model.train()
$\endgroup$
0
$\begingroup$

Okay... I made a stupid mistake... During validation I wrote:

# Move data to default device
lookback    = lookback.to(device)
attention   = attention.to(device)
openSta     = openSta.to(device)
currentopen = openSta.to(device)
promo       = promo.to(device)
labels      = openSta.to(device)

where there are multiple value mis-assigned... correct on should be:

lookback    = lookback.to(device)
attention   = attention.to(device)
openSta     = openSta.to(device)
currentopen = currentopen .to(device)
promo       = promo.to(device)
labels      = labels.to(device)

After the correction, the validation loss drop along with training loss. So I'll guess, except data issue, the valdation loss should always goes down with training loss within the first epoch...

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.