Pre-trained RNN for removing seasonality like VGG16

In image classification there are Pre-trained networks like VGG16. Are there any such networks for time series operations like removing seasonality?

Edit: I found the following paper TimeNet: Pre-trained deep recurrent neural network for time series classification, but is the implementation published? I'm planning to send the authors an email.

• by all means, please write to them, but I'd be extremely surprised if they shared the implementation with you. I'd be happy to be proven wrong, though. In that case, please let me know by writing a comment and citing me with @DeltaIV – DeltaIV Jun 6 '18 at 15:11
• ps if you just need to remove seasonality, I'd read a book such as Forecasting: principles and practices by @RobHyndman, rather than using an RNN. – DeltaIV Jun 6 '18 at 15:13
• yeah, good luck with "benefitting from the RNN hype". If you've never used RNNs and you want to beat "standard" methods, you're in for a hell of a good time: journals.plos.org/plosone/article?id=10.1371/… To be honest, the paper leaves a lot to be desired, but it does show a point: if you've never used RNNs, you'll have an hard time beating classical methods on short, univariate time series. This is not to speak ill of RNNs (I use them a lot), but the "barrier to entry" is higher than for CNNs. Why don't you describe your use case more in detail, so that 1/ – DeltaIV Jun 6 '18 at 16:59
• ps I haven't read it, but arxiv.org/pdf/1805.03908.pdf cites the paper you refer to, and you may find it useful. – DeltaIV Jun 6 '18 at 17:20
• if you're familiar with LSTMs, then things are of course different, but still it would be better if you could flesh out your question. Removing seasonality is pretty straightforward with usual time series methods or even with probabilistic programming tools such as stan (see Facebook's Prophet). As the question currently stands, there's no way to say if RNNs would be a good fit for you. – DeltaIV Jun 6 '18 at 17:39

1 Answer

While transfer learning enjoys widespread success for image classification with CNNs, it's much less common (and less successful) for time series forecast with RNNs. This paper seems to give a negative answer to the question "can Machine Learning methods beat "standard" time series forecast methods?":

http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0194889

The paper has a number of flaws:

• it uses the M3 dataset. Very short univariate time series are definitely not the setting for which RNN architectures such as LSTMs were born. A test on M4 would have been more interesting, even though we are again talking about univariate time series here. The first author of the paper actually wants to publish another paper, after the M4 competition has closed, showing the results of DL methods applied to M4. This would be good, but he should probably collaborate with someone with more experience on current Deep Learning best practices.
• the same preprocessing was used for all ML methods, chosen on the results for the MLP, with the justification that the MLP is "the most popular machine learning method". For time series prediction? Yeah, in the '90 maybe.
• their implementation of LSTMs is not clear at all. Why not sharing the code? They mention "all linear activation functions" except for one "hard sigmoid". The LSTM cell should have four gates, three using a sigmoid and one using a tanh activation function: no gate has a linear activation function. What architecture did they use, exactly?
• no use of GPUs, depth (1 hidden layer) or width (6 hidden units!). This could make sense, given the "small data" setting of M3, but then again, it would be expected that Deep Learning would fail in such a setting.
• no investigation about the batch size or the learning rate.

Thus, it doesn't exactly prove that RNNs are unsuitable for time series forecasting: assuming their implementation was not flawed, it shows that for small time series they don't work well "out of the box". If this is your setting, then avoid RNNs altogether.

There is another paper which could be quite interesting for you: I haven't had the time to read it in detail, but it tries to develop a "universal" time series classifier based on neural networks, i.e., a model which, once trained on a large data set, can be retrained with minimal transfer learning on data from a different domain and learn to classify time series from the new domain. This addresses classification and not forecasting, though. Also, it shows how hard is to match the level of SOTA time series classifiers (not NN-based) using NNs. Finally, it doesn't use RNNs but CNNs with attention.