3
$\begingroup$

I am trying to train a neural network on some time series data and decided to implement cross validation for my model.

The cross validation method I'm trying to implement is the Day Forward-Chaining from this post where each validation step uses more training data than the previous step.

For example:

1. train: 1, 2       -> val: 3, 4 -> test: 5, 6
2. train: 1, 2, 3, 4 -> val: 5, 6 -> test: 7, 8

As tensorflow requires its samples to be of the same size a sollution is to pad the smaller inputs as mentioned in How to feed input with changing size in Tensorflow.

What I would like to know is whether pre or post padding the sequences makes any difference. Would the model assume these are the same?

x, x, 1, 2 -> here the fist time step is now the third. Will the model be aware of this?
1, 2, x, x -> but here the padding occurs where the validation data should go.

Is there a proper way of doing this or does it not matter?

Another interesting question which I came accross when writing this is Padding - which values for standardized time series data?

What is the best value to use when padding a sequence? My data contains zeros so those should not be ignored. I though of -1 as data is never negative but standardizing it will change that.

$\endgroup$
2
  • $\begingroup$ What do you mean by "pre or post padding"? $\endgroup$ – Jindřich Oct 14 '20 at 7:51
  • $\begingroup$ @Jindřich Should I add the padding data before the first time step or after the last time step? I tried to explain it visually in the second box in the question (with the x's). $\endgroup$ – Marcus Oct 14 '20 at 7:56
2
$\begingroup$

The usual approach is to put the padding at the end of the sequence. It is more convenient for RNNs because it trivially ensures that the initial RNN state will the same everywhere.

You should keep separately a binary mask of what the valid positions are (and then it does not really matter what the pading values are). When you do the standardtization, you do not want to include the padding values anyway and you need to mask out the padding for computing the mean and variance.

If you plan to process the sequence with a Transformer, you will need the mask to tell the self-attention to what positions it can attend.

Finally, you will need the mask to compute the loss function, to be able base the loss function on the valid positions only.

$\endgroup$
2
  • $\begingroup$ I don't understand how the mask will compute the loss function if it is a matrix of binary values. If I give a mask and a loss function to tensorflow will it know what to do or do I have to adjust it some how? $\endgroup$ – Marcus Oct 14 '20 at 8:14
  • $\begingroup$ It depends on what predictions you want to make. If you make one prediction per input, then you want to consider only the valid inputs and therefore you need to do masking. $\endgroup$ – Jindřich Oct 15 '20 at 7:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.