0
$\begingroup$

I have several questions regarding Bi-RNN. The RNN here can be LSTM or GRU as well.

(1) What is the input of Bi-RNN when making inference? For RNN, if I want to predict a $\hat{y}(t)$ for the target $y(t)$, I need to input $x(1), x(2)...x(t-1)$ one by one recursively to get the state $h(t-1)$, then, together with $x(t)$, I can compute $\hat{y}(t)$. So the input needed for RNN is $$<x(1), x(2)...x(t-1), x(t)> \rightarrow y(t)$$ How about BI-RNN? Does the input require the sequence $x(1)$ to $x(t)$ for the RNN component with normal direction, and the sequence $x(t+n)$ to $x(t)$ for the RNN component with reverse direction?: $$<x(1), x(2)...x(t-1), x(t)>,<x(t), x(t+1)..., x(t+n)> \rightarrow y(t)$$

(2) If I also make a reverse copy of the original sequence data, and append it with the original sequence data, and then use the new dataset (size doubled) as the input to train an vanilla RNN, how is it different with Bi-RNN trained with the original data only? Because the new doubled data seems to me also contains bidirectional context relations for each target value.

What I mean is if we are training to predict $y(t)$, the input data can be two separated series, just like two lines of input data: $$<x(1), x(2)...x(t-1), x(t)> \rightarrow y(t)$$ $$<x(t+n), ..., x(t+1), x(t)> \rightarrow y(t)$$ It seems to me this still make sense in terms of the "time travel cycle".

One difference I notice is Bi-RNN doesnot share the parameters across the two RNN components. But the vanilla RNN does. Also, it seems the RNN with the doubled data can be trained, but can not be used to predict with the full context (pre- & post- contexts)? Because the input is only one directional. We cannot input both the pre- and post- context at the same time to predict $y(t)$. Is that the reason why we do not use the vanilla RNN with the doubled data (both original data and reverse data) as training data?

$\endgroup$
0
$\begingroup$

With RNN going in left-to-right direction you model sequence

$$ x(1), x(2), \dots, x(t-1), x(t) $$

with right-to-left RNN, it goes backwards in time

$$ x(t), x(t-1), \dots, x(2), x(1) $$

In bi-directional RNN, we compute both and combine them.

What you are proposing is to model the sequence

$$ x(1), x(2), \dots, x(t-1), x(t), x(t), x(t-1), \dots, x(2), x(1) $$

If you look closely, now the "second" $x(t-1)$ depends on $x(t), x(t), x(t-1), \dots$ and this doesn't make much sense, since you are introducing some kind of time travel cycle from the future to the past and dependence of $x(t-1)$ on itself.

$\endgroup$
  • $\begingroup$ Hi Tim. Thanks for the reply. I think I didn't make it clear in my question. What I mean is the input data can be two series instead of one concatenated series: $$<x(1), x(2)...x(t-1), x(t)> \rightarrow y(t)$$ $$<x(t+n), ..., x(t+1), x(t)> \rightarrow y(t)$$ It seems to me this still make sense in terms of the "time travel cycle". $\endgroup$ – Tyler 十三将士归玉门 Mar 25 at 5:15
  • $\begingroup$ @Tyler十三将士归玉门 So you are using different series for forward and backward steps..? If yes, this is not bidirectional... $\endgroup$ – Tim Mar 25 at 6:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.