Interpreting gradient descent as a constrained optimization problem- Reinforcement learning

Question

I' m studying the lectures of Sergey Levine in reinforcement learning, specifically the TRPO algorithm, during his explanation we claims that gradient descent is the same as doing this.

He does not show the proof to why that is. I intuitively understand that we have a local linearization of the objective function and we can only use that approximation arround $\theta$, therefore using the euclidean distance between $\theta$ and $\theta'$ and limiting by $\epsilon$ we guarantee that we don't update the parameters to much to the point where the approximation has less precision.

Can someone help figure out how he gets the result for $\theta'$. I tried to construct the Lagragian an setting its' gradient to zero, but I got lost in the computation.

Thank you very much in advance.

Answer 1

I think its simpler than it looks, from proposed gradient equation:

$\theta' = \theta + \frac{\epsilon}{||\nabla_\theta J(\theta)||^2} \nabla_\theta J(\theta)$

we get

$\theta' - \theta= \frac{\epsilon}{||\nabla_\theta J(\theta)||^2} \nabla_\theta J(\theta)$

if we plug that into

$\theta' \leftarrow \arg \max_\limits{\theta'} \nabla_\theta J(\theta)^T (\theta - \theta')$

we get

$\theta' \leftarrow \arg \max_\limits{\theta'} \nabla_\theta J(\theta)^T \frac{\epsilon}{||\nabla_\theta J(\theta)||^2} \nabla_\theta J(\theta)$

$\nabla_\theta J(\theta)^T \nabla_\theta J(\theta)$ gives $||\nabla_\theta J(\theta)||^2$ and those two terms cancel.
Now we have:

$\theta' \leftarrow \arg \max_\limits{\theta'} \epsilon$

$\theta' \leftarrow \arg \max_\limits{\theta'} ||\theta - \theta'||^2$

with this we showed that by doing the proposed gradient update we get the vector $\theta'$ that maximizes the Euclidian distance with given constraint