3
$\begingroup$

Suppose I'm trying to predict time series with a neural network. The data set is created from a single column of temporal data, where the inputs of each pattern are [t-n, t-n+1, ... , t], t being the time step and n the embedding size, and [t+1] being the target (predicting the "next step" of the series). Here is the question: if I use such a data set for NN training, should I shuffle it? Shuffling is usually done to avoid overfitting by preventing the NN from memorizing the pattern order. However, in case of time series, could pattern order be a form of information that I shouldn't deprive the NN of? The issue becomes even more severe if recurrent NNs are concerned, where every pattern recurrently receives information about the previous pattern: would a RNN benefit if patterns were in order?

$\endgroup$
3
$\begingroup$

Say you have a single time series you want to learn on. Then you can use the first half for the development of your model and the second half for testing.

You now cut your two halfs into windows individually and can shuffle those of the training set.

From what I know, the shuffling is actually not because of generalization, but because of optimization. It is sometimes more efficient to optimize a sum of functions (in this case, one loss function per time window) if you do not look at their sum, but estimate the gradient by looking at the gradients of a subset of the sum. Look at recent publications by Nicolas Le Roux and Marc Schmidt for this topic.

$\endgroup$
  • $\begingroup$ Thanks, I'll check it out. Maybe I just had a wrong idea stuck in my head all along the road :) $\endgroup$ – anna-earwen Oct 25 '12 at 11:03
  • $\begingroup$ Hmm... Now, that only applies to gradient-based learning algorithms. I.e., if I use a population-based approach such as PSO or GA, shuffling may be completely irrelevant. $\endgroup$ – anna-earwen Oct 25 '12 at 11:05
  • $\begingroup$ It relates whether you are doing online learning, e.g. you estimate your loss or its gradient by only looking at a fraction. Nothing stops you from doing that with population based algorithms as well. However, you are right, this is mostly done for gradient based optimization only. $\endgroup$ – bayerj Oct 28 '12 at 19:37
  • $\begingroup$ Even for on-line learning, if past patterns are revisited, they should probably be shuffled. I carried out some experiments a few days ago and saw what I expected to see: backpropagation performed worse without shuffling, and PSO wasn't really affected (although I have to say no statistical significance tests were carried out... yet). $\endgroup$ – anna-earwen Oct 29 '12 at 20:22
  • $\begingroup$ That's what I'm saying, sorry. In the case of offline learning, where your loss is exact wrt training set, you don't have to shuffle. If you shuffle, you will get the same number. This is not the case for online algorithms, where you only look at a fraction of your training set and thus just an estimate of your training loss. $\endgroup$ – bayerj Oct 30 '12 at 7:07
1
$\begingroup$

If you are using a not-recurrent NN like a traditional MLP you don't NEED to shuffle dataset especially if you're using a batch learning algorithm. Anyway, in my experience it's a good idea to shuffle datasets for a faster training and more clear results. I can suggest you "Efficient BackProp" by LeCun et al., where you can find a useful set of tips & tricks on the use of NNs.

$\endgroup$
  • $\begingroup$ Thanks! I have a copy of that paper, I guess it's time to refresh my memory. $\endgroup$ – anna-earwen Oct 25 '12 at 11:01
  • 1
    $\begingroup$ I think that this is a dangerous answer. If you do not shuffle your data set and do stochastic gradient descent, you might get weird oscillating behaviour where you run in circles. Basically, you might follow a fixed trajectory through function space and follow it again and again. Actually, there is absolutely no reason to NOT shuffle your data set if it is finite. The only reason not to is if it has no effect, which is the case in batch learning. Otherwise, always shuffle since it does no harm at all. Another exception to this might be if you try to learn a non stationary function online. $\endgroup$ – bayerj Oct 29 '12 at 18:36
  • $\begingroup$ In the context of SGD, data should be shuffled. Otherwise, the gradient could lead to local minima and the learning curve can look weird. I have made this mistake before. $\endgroup$ – Jiang Xiang Aug 22 '16 at 16:29
  • 1
    $\begingroup$ I think this advice apply to the era before Recurrent Neural Network. My intuition tells me that braking the order you won't have the memory benefits of a RNN, unless you have an index as a feature. Please let me know your thoughts. $\endgroup$ – Germán Alfaro Feb 14 '18 at 0:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.