Skip to main content

Questions tagged [policy-iteration]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
11 views

How to modify the bellman operator for in-place iterative policy evaluation?

The iterative update rule for policy evaluation that is, approximating the value function for a given policy is: $$v^{k+1} = r_{\pi} + \gamma P_{\pi}v^{k}$$ This is the simultaneous update rule where ...
Atharva's user avatar
2 votes
1 answer
104 views

Is TRPO just a "safe" version of off-policy policy iteration

So, the constrained TRPO objective is the following: $$ J(\theta) = E_t\left[ \frac{\pi_\theta(a_t|s_t)}{\pi_{old}(a_t|s_t)}\cdot A_t \right]\\ st. D_{KL}[\pi_{old}(\cdot|s_t)||\pi(\cdot|s_t)] \le \...
Alberto's user avatar
  • 1,207
2 votes
1 answer
454 views

Understanding Policy Iteration and Value Iteration

I have read an answer there but still cannot capture the ideas behind it. In Sutton's book, section Value Iteration, it is said that In fact, the policy evaluation step of policy iteration can be ...
k2pctdn's user avatar
  • 23
1 vote
1 answer
143 views

Question on Equation 5.2 of Reinforcement Learning by Sutton and Barto

I'm currently studying the textbook Reinforcement Learning by Sutton and Barto. I can't seem to understand the derivation in Equation 5.2: How did (a) become (b)? In particular, why is the ...
prperalta's user avatar
3 votes
1 answer
335 views

Why the $\gamma^t$ is needed here in REINFORCE: Monte-Carlo Policy-Gradient Control (episodic) for $\pi_{*}$?

While reading PG method in Prof Sutton's RL book again, I found there is $\gamma^t$ in the last row (as shown below) in pseudo code. The book said The second difference between the pseudocode update ...
GoingMyWay's user avatar
  • 1,381
2 votes
1 answer
862 views

Policy improvement in SARSA and Q learning

I have a rather trivial doubt in SARSA and Q learning. Looking at the pseudocode of the two algorithms in Sutton&Barto book, I see the policy improvement step is missing. How will I get the ...
Jor_El's user avatar
  • 393
1 vote
1 answer
365 views

One small confusion on $\epsilon$-Greedy policy improvement based on Monte Carlo

I'm working on the RL book of Barto and Sutton, the author has provided the proof based on the policy improvement theorem, I can fully understand the inequality, but for the first equality, it really ...
FantasticAI's user avatar
4 votes
0 answers
832 views

Convergence Proof of First Visit Monte Carlo Control

I am currently trying to find a formal proof of convergence for the Monte Carlo Reinforcement Learning Methods described in Sutton,Barto's Book "Reinforcement Learning - An Introduction" , Section 5. ...
GreenLogic's user avatar
1 vote
0 answers
85 views

How to increase the total number of iterations it takes to converge a MDP?

I was reading about Policy Iteration. What are the factors that influence the total number of iterations the algorithm takes to converge? For a given MDP which converges in 3 iterations, what setting ...
Amanda's user avatar
  • 111
1 vote
1 answer
2k views

Q-learning shows worse results than value iteration

I'm trying to solve the same problem with different algorithms (Travel max possible distance with a car). While using value iteration and policy iteration I was able to get the best results possible ...
Most Wanted's user avatar
10 votes
1 answer
2k views

Why are the value and policy iteration dynamic programming algorithms?

Algorithms like policy iteration and value iteration are often classified as dynamic programming methods that try to solve the Bellman optimality equations. My current understanding of dynamic ...
Karthik Thiagarajan's user avatar
14 votes
3 answers
14k views

Why does the policy iteration algorithm converge to optimal policy and value function?

I was reading Andrew Ng's lecture notes on reinforcement learning, and I was trying to understand why policy iteration converged to the optimal value function $V^*$ and optimum policy $\pi^*$. ...
Charlie Parker's user avatar