1
$\begingroup$

Let's say I have a system, where given a current state $s$ and my action $a$ I can sample successor states $s'$. I am given a policy $\pi$ such that $\pi(s,a)$ is the probability of action $a$ taken in state $s$.

Let's say, I would like to evaluate $V^\pi$ using this sampler. That's easy, at least in theory: I start with a random $s$, always pick action according to current state and $\pi$, and sample until converge till the end of the episode. I get total reward $R$ over that episode, and then I update $V^\pi(s)$ with the average of all total rewards I got when I started from $s$. This is first-visit Monte Carlo as explained in Sutton and Barto "Reinforcement learning". Here I am not asking about computationally optimal methods, only about concept itself. I assume non-discounted additive reward case, and episodes, like in playing tic-tac-toe, for example.

Now, the question. How would I compute $Q^\pi(s,a)$ using the method above? My confusion comes from the fact that $\pi$ may assign different actions to $s$, with non-zero probability for actions $a'\neq a$ as well. So, whenever I sample, should I only apply $a$ when I am in state $s$, or should I only apply $a$ at $s$ at the very start, and then apply action at $s$ according to $\pi$? My guess is that the latter is the way to do. Is that right, and if it is - is it always the way I should think about $Q^\pi(s,a)$?

$\endgroup$
0
$\begingroup$

If you average all the returns, then you are doing every-visit MC; first-visit MC averages only the first return over all episodes.

Calculating action values works the same way as state values: you follow your policy and record the reward for the action you take at each state. You just have to make sure to choose a policy that explores every action.

$\endgroup$
0
$\begingroup$

Yes, $Q_\pi (s, a)$ is computed by taking action $a$ on the current time step, and acting according to $\pi$ thereafter, even on subsequent visits to state $s$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.