1
$\begingroup$

I'm trying to wrap my head around the finer points of backpropagation through time and was hoping to get some clarification on something. For the most part, I understand the general idea of BPTT: we feed inputs forward through the RNN to estimate output, send the errors backwards along the network to calculate gradients of the loss function w.r.t. the various parameters, and iterate this process until convergence. Then there's the minibatch/stocastic version of gradient descent, in which we repeatedly use random subsets of the training data (as opposed to all of the data at once) to get our updates.

I get that, but most of the resources I can find present the equations assuming a single training sequence. In general, the training data for an RNN can be 3-dimensional: [number of sequences, time steps, number of features]. When we talk about minibatches, are we sampling time steps (i.e. sampling along the second dimension), or sequences (sampling along the first dimension)?

Also, how does the math behind BPTT change with multiple training sequences? Is it simply a matter of double-summing loss contributions over both time-steps and sequences?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.