2
$\begingroup$

If a single output is predicted by an RNN at the end of a timeseries, frequently the outputs of each timestep of the RNN are averaged to make this prediction. My impression is that it seems strange, but it works.

Suppose an output needs to be predicted for each of n timesteps. Does it therefore make sense to use the average of outputs 1,..,k for prediction k<=n so as to generalize the above?

I would be interested if anyone has convincing evidence or opinions one way or the other.

$\endgroup$
0
$\begingroup$

See Figure 10.5 of Deep Learning (Goodfellow, Bengio, Courville). The architecture itself can be adapted to fit this use case. If there are no labels for each time step (just for the last), I believe this is necessary (although you can also replicate the labels (target replication)). Target replication is performed in Lipton's 2016 paper on learning to diagnose with LSTMs.

It seems to me (just an opinion) that some kind of target replication (or, if you have them, adding extra labels) might work better. In training a many-to-one, you have to send the gradient backwards a long distance, which can cause issues. More frequent targets give more immediate feedback on how adjacent-to-target weights influence the loss.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.