I need advice to model a certain kind of time series prediction for which I didn't find any existing solution. I have a large set of independent time series of fixed length (let's say 100 steps). All those series
- Start with y=0
- Grow continuously
- Share some time-dependent patterns
At prediction time I get as input a new incomplete series that could end at any point and I need the complete continuation until the end. This greatly simplified example illustrates what I mean:
There are multiple features, some of them are time dependent and some are not (the latter only providing information about a global scale, e.g. an estimation of the final value).
A lot of what I read about RNNs isn't applicable here, for example because of the fixed length output length. So far I trained some LSTM models using a many-to-one architecture, where the next gain upwards is the output value. I then recursively use the predictions as input for the next step prediction until I arrive at the end (I can derive and update the other time dependent features in that process).
The trainings converge quite well but the resulting predictions are mediocre and kind of chaotic. Only small changes in the input can make the difference between a reasonable continuation and total nonsense.
Another problem: Some of the patterns of the true series only occur at certain times, for this reason I use the time step number as input feature. But the model often reproduces those patterns at completely wrong times.
Someone has an idea for a better approach?
e.: I cannot train one model per time step because actually there are many more than 100 and I wouldn't have enough data for each one.