While transfer learning enjoys widespread success for image classification with CNNs, it's much less common (and less successful) for time series forecast with RNNs. This paper seems to give a negative answer to the question "can Machine Learning methods beat "standard" time series forecast methods?":
The paper has a number of flaws:
- it uses the M3 dataset. Very short univariate time series are definitely not the setting for which RNN architectures such as LSTMs were born. A test on M4 would have been more interesting, even though we are again talking about univariate time series here. The first author of the paper actually wants to publish another paper, after the M4 competition has closed, showing the results of DL methods applied to M4. This would be good, but he should probably collaborate with someone with more experience on current Deep Learning best practices.
- the same preprocessing was used for all ML methods, chosen on the results for the MLP, with the justification that the MLP is "the most popular machine learning method". For time series prediction? Yeah, in the '90 maybe.
- their implementation of LSTMs is not clear at all. Why not sharing the code? They mention "all linear activation functions" except for one "hard sigmoid". The LSTM cell should have four gates, three using a sigmoid and one using a tanh activation function: no gate has a linear activation function. What architecture did they use, exactly?
- no use of GPUs, depth (1 hidden layer) or width (6 hidden units!). This could make sense, given the "small data" setting of M3, but then again, it would be expected that Deep Learning would fail in such a setting.
- no investigation about the batch size or the learning rate.
Thus, it doesn't exactly prove that RNNs are unsuitable for time series forecasting: assuming their implementation was not flawed, it shows that for small time series they don't work well "out of the box". If this is your setting, then avoid RNNs altogether.
There is another paper which could be quite interesting for you: I haven't had the time to read it in detail, but it tries to develop a "universal" time series classifier based on neural networks, i.e., a model which, once trained on a large data set, can be retrained with minimal transfer learning on data from a different domain and learn to classify time series from the new domain. This addresses classification and not forecasting, though. Also, it shows how hard is to match the level of SOTA time series classifiers (not NN-based) using NNs. Finally, it doesn't use RNNs but CNNs with attention.