1
$\begingroup$

I need to track the computation and storage of different parts of my network training. To be on the same page, let's assume the simple following scenario (biases omitted) enter image description here

Questions

Local Gradients - Are the intermediate gradients always stored in memory until they are consumed in the backward pass, being eliminated after that? Or is there an option to not storing but recompute them when needed during the backward pass? - How can I access those values right after they are computed in the forward pass? Is there anything similar to self.layer_i.weight.data.numpy()?

Back-Propagated Error - Is it right the theoretical understanding of back-propagated error? In the Glorot and Bengio paper (equations 13 and 14 in particular) they make a different analysis for weight gradients and back-propagated gradients and I want to make sure I got that right before analyzing anything. - Is it possible again to access directly these values from the networks attributes? Something like required_grad = True that could allow me to use self.layer_i.weight.data.numpy() for the input instead of for the parameters (weights)?

Thank you

$\endgroup$
1
$\begingroup$

I can answer the first question:

Recomputing gradients to save memory makes sense. This is usually called "gradient checkpointing". Here is one such paper, although there are many. You pay a small time penalty for a large savings in memory. The key is not to recompute all the gradients, but rather to save the gradients at strategic "checkpoint" layers in the computation graph, so that you only ever need to recompute a small number of gradients.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.