1
$\begingroup$

A typical way of implementing mini batch learning is by calculating the gradients of every element within the mini batch and then average all of these element's gradients to come up with the final gradient.

Gradient clipping occurs after we have averaged out all the gradients within the mini batch.

My question is, what if we performed gradient clipping on every single gradient in the mini batch, and then average out the clipped gradients. Would this make any sense? From a bias-variance point of view would it be the same as the typical way of doing gradient clipping?

$\endgroup$
1
  • $\begingroup$ If the relationship between the dependent variable (DV) and any of the independent variables (IV) is not linear, for example a logarithmic or sine (seasonal) relationship, the gradient clipping will likely introduce severe errors in the modeling results. One way to prevent this is to visually inspect scatterplots of each IV versus the DV to determine if any such relationship is present, in which case it might be possible to mitigate the problem by data transformation prior to regression. $\endgroup$ May 25 '19 at 13:57
0
$\begingroup$

The mean value you will obtain by averaging clipped individual observations is similar to truncated mean. Yet, truncated mean is obtained by symmetrical trimming (same percentage removed from both upper and lower tails). In contrast, gradient clipping is asymmetrical and accordingly yields skewed individual observations. According to Wikipedia (link):

... unless the underlying distribution is symmetric, the truncated mean of a sample is unlikely to produce an unbiased estimator for either the mean or the median.

Then, your mean gradient estimate will be biased if clipping is applied on each individual prior to averaging.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.