1
$\begingroup$

I just started learning RNNs. I'm stuck with two basic doubts:

1.)We start with same weights for each timestamp; loss function can be the logistic loss or squared error across timestamps. In truncated BPTT, I assume we only unroll k timestamps back at each timestamp for calculating the partial derivative of loss with respect to weights. So for calculating partial derivate of loss w.r.t, last time stamp activation involves calculating partial derivative w.r.t previous k timestamps. So I assume we first start calculating partial derivatives of loss from the first stamp so they can be used at the later stage. Please correct me if I'm stating some obvious erroneous statements.

2.) Do we reset weights and perform BPPT for each training example?

credits:@ ragulpr

$\endgroup$
0
$\begingroup$

So for calculating partial derivate of loss w.r.t, last time stamp activation involves calculating partial derivative w.r.t previous k timestamps.

Yes. At time $t+1$ your cell gets input element of sequence and hidden state from time $t$. That means that you need use derivative of the function that gets this state to compute derivative at state $t+1$

2.) Do we reset weights and perform BPPT for each training example?

No, because that would mean you don't learn anything. Fitting NN is called learning, because in each batch/example you update weights. If you just started from scratch every time your model wouldn't learn anything.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.