# Questions tagged [policy-iteration]

The tag has no usage guidance.

12 questions
Filter by
Sorted by
Tagged with
11 views

### How to modify the bellman operator for in-place iterative policy evaluation?

The iterative update rule for policy evaluation that is, approximating the value function for a given policy is: $$v^{k+1} = r_{\pi} + \gamma P_{\pi}v^{k}$$ This is the simultaneous update rule where ...
104 views

### Is TRPO just a "safe" version of off-policy policy iteration

So, the constrained TRPO objective is the following:  J(\theta) = E_t\left[ \frac{\pi_\theta(a_t|s_t)}{\pi_{old}(a_t|s_t)}\cdot A_t \right]\\ st. D_{KL}[\pi_{old}(\cdot|s_t)||\pi(\cdot|s_t)] \le \...
• 1,207
454 views

### Understanding Policy Iteration and Value Iteration

I have read an answer there but still cannot capture the ideas behind it. In Sutton's book, section Value Iteration, it is said that In fact, the policy evaluation step of policy iteration can be ...
• 23
1 vote
143 views

### Question on Equation 5.2 of Reinforcement Learning by Sutton and Barto

I'm currently studying the textbook Reinforcement Learning by Sutton and Barto. I can't seem to understand the derivation in Equation 5.2: How did (a) become (b)? In particular, why is the ...
335 views

### Why the $\gamma^t$ is needed here in REINFORCE: Monte-Carlo Policy-Gradient Control (episodic) for $\pi_{*}$?

While reading PG method in Prof Sutton's RL book again, I found there is $\gamma^t$ in the last row (as shown below) in pseudo code. The book said The second difference between the pseudocode update ...
• 1,381
862 views

### Policy improvement in SARSA and Q learning

I have a rather trivial doubt in SARSA and Q learning. Looking at the pseudocode of the two algorithms in Sutton&Barto book, I see the policy improvement step is missing. How will I get the ...
• 393
1 vote
365 views

### One small confusion on $\epsilon$-Greedy policy improvement based on Monte Carlo

I'm working on the RL book of Barto and Sutton, the author has provided the proof based on the policy improvement theorem, I can fully understand the inequality, but for the first equality, it really ...
• 477
832 views

### Convergence Proof of First Visit Monte Carlo Control

I am currently trying to find a formal proof of convergence for the Monte Carlo Reinforcement Learning Methods described in Sutton,Barto's Book "Reinforcement Learning - An Introduction" , Section 5. ...
• 203
1 vote
85 views

### How to increase the total number of iterations it takes to converge a MDP?

I was reading about Policy Iteration. What are the factors that influence the total number of iterations the algorithm takes to converge? For a given MDP which converges in 3 iterations, what setting ...
• 111
1 vote
2k views

### Q-learning shows worse results than value iteration

I'm trying to solve the same problem with different algorithms (Travel max possible distance with a car). While using value iteration and policy iteration I was able to get the best results possible ...
• 275
I was reading Andrew Ng's lecture notes on reinforcement learning, and I was trying to understand why policy iteration converged to the optimal value function $V^*$ and optimum policy $\pi^*$. ...