7
$\begingroup$

I am reading these notes on slide 34 and came across strategies to prevent gradients from becoming too big in Deep Q Learning (DQN). Since, we don't usually use deep architectures in DQN, I don't think it's an exploding gradient problem. My understanding is that it has something to do with the linear regression squared error loss function, since DQN is a regression network. Could someone please explain it to me?

$\endgroup$
2
  • 2
    $\begingroup$ Since we don't usually use deep architectures in DQN I don't think this is true, AlphaGo - the most well known example, used a deep q network. $\endgroup$ Jul 27, 2022 at 5:49
  • $\begingroup$ Thank you for your comment. I did qualify my sentence with "usually" :) Most RL papers in my field, benchmark their code using Atari/Gym environments. They usually have shallow networks. $\endgroup$ Jul 27, 2022 at 14:36

1 Answer 1

7
$\begingroup$

It's hard to say for sure, since the slide is so sparse, but I think the implication is that squaring a number with absolute value larger than 1 can become very large very quickly, and likewise the gradients can grow very fast. The author's suggestion of Huber loss makes sense as a remedy to this problem, because you can choose where the loss becomes MAE loss, which does not grow as quickly as a square error loss.

Likewise, this suggests that you're correct that the problem isn't exploding gradients. Exploding gradients refers to multiple layers having gradients larger than 1, so the magnitude of the update compounds as the chain rule is applied.

Even though the gradient is not exploding, it's still true that gradients can cause instability in training -- large steps in nearly-orthogonal directions can prevent the model from improving, or simply slow improvement.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.