Questions tagged [policy-iteration]
The policy-iteration tag has no usage guidance.
12 questions
0
votes
0
answers
21
views
How to modify the bellman operator for in-place iterative policy evaluation?
The iterative update rule for policy evaluation that is, approximating the value function for a given policy is:
$$v^{k+1} = r_{\pi} + \gamma P_{\pi}v^{k}$$
This is the simultaneous update rule where ...
- 1
14
votes
3
answers
15k
views
Why does the policy iteration algorithm converge to optimal policy and value function?
I was reading Andrew Ng's lecture notes on reinforcement learning, and I was trying to understand why policy iteration converged to the optimal value function $V^*$ and optimum policy $\pi^*$.
...
- 101
2
votes
1
answer
140
views
Is TRPO just a "safe" version of off-policy policy iteration
So, the constrained TRPO objective is the following:
$$
J(\theta) = E_t\left[
\frac{\pi_\theta(a_t|s_t)}{\pi_{old}(a_t|s_t)}\cdot A_t
\right]\\
st. D_{KL}[\pi_{old}(\cdot|s_t)||\pi(\cdot|s_t)] \le \...
- 1,335
2
votes
1
answer
522
views
Understanding Policy Iteration and Value Iteration
I have read an answer there but still cannot capture the ideas behind it.
In Sutton's book, section Value Iteration, it is said that
In fact, the policy evaluation step of policy iteration can be ...
- 1,335
1
vote
1
answer
170
views
Question on Equation 5.2 of Reinforcement Learning by Sutton and Barto
I'm currently studying the textbook Reinforcement Learning by Sutton and Barto. I can't seem to understand the derivation in Equation 5.2:
How did (a) become (b)?
In particular, why is the ...
- 1,335
11
votes
1
answer
3k
views
Why are the value and policy iteration dynamic programming algorithms?
Algorithms like policy iteration and value iteration are often classified as dynamic programming methods that try to solve the Bellman optimality equations.
My current understanding of dynamic ...
CommunityBot
- 1
3
votes
1
answer
364
views
Why the $\gamma^t$ is needed here in REINFORCE: Monte-Carlo Policy-Gradient Control (episodic) for $\pi_{*}$?
While reading PG method in Prof Sutton's RL book again, I found there is $\gamma^t$ in the last row (as shown below) in pseudo code. The book said
The second difference between the pseudocode update ...
- 83
2
votes
1
answer
915
views
Policy improvement in SARSA and Q learning
I have a rather trivial doubt in SARSA and Q learning. Looking at the pseudocode of the two algorithms in Sutton&Barto book, I see the policy improvement step is missing.
How will I get the ...
- 6,944
1
vote
1
answer
393
views
One small confusion on $\epsilon$-Greedy policy improvement based on Monte Carlo
I'm working on the RL book of Barto and Sutton, the author has provided the proof based on the policy improvement theorem, I can fully understand the inequality, but for the first equality, it really ...
- 7,031
4
votes
0
answers
885
views
Convergence Proof of First Visit Monte Carlo Control
I am currently trying to find a formal proof of convergence for the Monte Carlo Reinforcement Learning Methods described in Sutton,Barto's Book "Reinforcement Learning - An Introduction" , Section 5.
...
- 203
1
vote
0
answers
88
views
How to increase the total number of iterations it takes to converge a MDP?
I was reading about Policy Iteration. What are the factors that influence the total number of iterations the algorithm takes to converge?
For a given MDP which converges in 3 iterations, what setting ...
- 111
1
vote
1
answer
2k
views
Q-learning shows worse results than value iteration
I'm trying to solve the same problem with different algorithms (Travel max possible distance with a car). While using value iteration and policy iteration I was able to get the best results possible ...
- 6,944