I have a layer $f_{(a,b)}$, where $(a,b)$ are some parameters. During training, $(a,b)$ get updated using a custom update-scheme $g$.

The thing is that $(a,b)$ don't get updated during the forward-pass, but during the backward-pass, since otherwise, it would allow the network to cheat on my task (by using knowledge about the batch it should otherwise generalize over).

A simple example: Let's say $(a,b)$ are mean and variance, which I update using an exponential-decayed estimator.

My problem now is that I encounter unstable training, which I assume is due to my gradient not being correct anymore. Is there some general formula on how to correct my gradients when custom update-schemes are present in the network?

To illustrate my problem, let's again assume I have the weird batchnorm-variant mentioned above. If my gradients from the previous layers point towards a too-high bias-value in my previous layers and i pass through my $f$, I would assume I would have to correct the gradient for the updated bias and variance (though subtracting and scaling).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.