0
$\begingroup$

I have a layer $f_{(a,b)}$, where $(a,b)$ are some parameters. During training, $(a,b)$ get updated using a custom update-scheme $g$.

The thing is that $(a,b)$ don't get updated during the forward-pass, but during the backward-pass, since otherwise, it would allow the network to cheat on my task (by using knowledge about the batch it should otherwise generalize over).

A simple example: Let's say $(a,b)$ are mean and variance, which I update using an exponential-decayed estimator.

My problem now is that I encounter unstable training, which I assume is due to my gradient not being correct anymore. Is there some general formula on how to correct my gradients when custom update-schemes are present in the network?

To illustrate my problem, let's again assume I have the weird batchnorm-variant mentioned above. If my gradients from the previous layers point towards a too-high bias-value in my previous layers and i pass through my $f$, I would assume I would have to correct the gradient for the updated bias and variance (though subtracting and scaling).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.