Do RNNs adjust memory during prediction/forward pass?

Question

Let's say i train a RNN to predict some curve given only 1 timestep as input. So the input is x(t) and the target is x(t+1). On the next timestep the input is x(t+1) and the target is x(t+2) and so on. During training it can memorize the inputs from previous timesteps and thus make more accurate predictions and afterwards the RNN can accurately predict the curve.

What i don't understand: If i have finished training and start predicting on a random point on the curve, how can the RNN predict anything? The memory it has is from the training data and maybe we stopped training when the curve was going up, but now we are predicting somewhere where it goes down. Is the RNN able to adjust its memory during prediction? So we could let it run for say 10 timesteps by using the available data and afterwards it can start using the predictions as input again?

If the question is unclear feel free to tell me and I will try to make it clearer.

ps: I asked the same question on reddit but haven't gotten an answer so far.

Answer 1

Usually, we start off each sequence with a special input. This will allow the RNN to initialize it's memory cells. Then, it can generate as many new samples as you want starting from the start state.

The memory cells do not carry over from training time to test time. The model learns the properties of the training data and encodes it into the parameters of the RNN, not the memory.

Answer 2

You feed the model a sequence of inputs at test time and let it finish it, rather than just one input. Take for example a model trained on text. If you just give it the letter "t", the next letter could really be almost anything. But if you give it the letters "this pie is goo" it will be likely to predict the next letter being "d". And the same goes with a curve.

Answer 3

Yes, you should refeed the predictions if you want to predict further into the future. However, with a one delay only model you'll never be able to detect/predict ups and downs. This is because a NN with only one input is simply a logistic regression (if you use sigmoid) and therefore is a linear model, so predictions will either go up, down or straight. You'll need at least two inputs if you want your NN to detect ups and downs, because then maybe up-down predicts down again, down down predicts up and so on. I'm simplifying here, but I hope you get the picture.