# Correct gradient with custom weight update

I have a layer $$f_{(a,b)}$$, where $$(a,b)$$ are some parameters. During training, $$(a,b)$$ get updated using a custom update-scheme $$g$$.

The thing is that $$(a,b)$$ don't get updated during the forward-pass, but during the backward-pass, since otherwise, it would allow the network to cheat on my task (by using knowledge about the batch it should otherwise generalize over).

A simple example: Let's say $$(a,b)$$ are mean and variance, which I update using an exponential-decayed estimator.

My problem now is that I encounter unstable training, which I assume is due to my gradient not being correct anymore. Is there some general formula on how to correct my gradients when custom update-schemes are present in the network?

To illustrate my problem, let's again assume I have the weird batchnorm-variant mentioned above. If my gradients from the previous layers point towards a too-high bias-value in my previous layers and i pass through my $$f$$, I would assume I would have to correct the gradient for the updated bias and variance (though subtracting and scaling).