I' m studying the lectures of Sergey Levine in reinforcement learning, specifically the TRPO algorithm, during his explanation we claims that gradient descent is the same as doing this. enter image description here

He does not show the proof to why that is. I intuitively understand that we have a local linearization of the objective function and we can only use that approximation arround $\theta$, therefore using the euclidean distance between $\theta$ and $\theta'$ and limiting by $\epsilon$ we guarantee that we don't update the parameters to much to the point where the approximation has less precision.

Can someone help figure out how he gets the result for $\theta'$. I tried to construct the Lagragian an setting its' gradient to zero, but I got lost in the computation.

Thank you very much in advance.


1 Answer 1


I think its simpler than it looks, from proposed gradient equation:

$\theta' = \theta + \frac{\epsilon}{||\nabla_\theta J(\theta)||^2} \nabla_\theta J(\theta)$

we get

$\theta' - \theta= \frac{\epsilon}{||\nabla_\theta J(\theta)||^2} \nabla_\theta J(\theta)$

if we plug that into

$\theta' \leftarrow \arg \max_\limits{\theta'} \nabla_\theta J(\theta)^T (\theta - \theta')$

we get

$\theta' \leftarrow \arg \max_\limits{\theta'} \nabla_\theta J(\theta)^T \frac{\epsilon}{||\nabla_\theta J(\theta)||^2} \nabla_\theta J(\theta)$

$\nabla_\theta J(\theta)^T \nabla_\theta J(\theta)$ gives $||\nabla_\theta J(\theta)||^2$ and those two terms cancel.
Now we have:

$\theta' \leftarrow \arg \max_\limits{\theta'} \epsilon$

$\theta' \leftarrow \arg \max_\limits{\theta'} ||\theta - \theta'||^2$

with this we showed that by doing the proposed gradient update we get the vector $\theta'$ that maximizes the Euclidian distance with given constraint

  • $\begingroup$ First of all let me thank you for spending the time to give a hand! Now i can see how moving $\frac{\epsilon}{\left \| \nabla {J(\theta)} \right \|^{2}} $in the direction of the gradient, limits the change in $\theta'$ so that $\left\| \theta - \theta '\right\|\leq \epsilon $. Can the lagragian approach work as well, if done right? $\endgroup$ Commented Mar 21, 2019 at 15:55
  • $\begingroup$ possibly, I'm not sure, I couldn't really help you with that, if you're really interested you could ask on math stack exchange site which is more appropriate for strictly mathematical questions, you could get better help there with that $\endgroup$
    – Brale
    Commented Mar 21, 2019 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.