Let's say I have a system, where given a current state $s$ and my action $a$ I can sample successor states $s'$. I am given a policy $\pi$ such that $\pi(s,a)$ is the probability of action $a$ taken in state $s$.
Let's say, I would like to evaluate $V^\pi$ using this sampler. That's easy, at least in theory: I start with a random $s$, always pick action according to current state and $\pi$, and sample until converge till the end of the episode. I get total reward $R$ over that episode, and then I update $V^\pi(s)$ with the average of all total rewards I got when I started from $s$. This is first-visit Monte Carlo as explained in Sutton and Barto "Reinforcement learning". Here I am not asking about computationally optimal methods, only about concept itself. I assume non-discounted additive reward case, and episodes, like in playing tic-tac-toe, for example.
Now, the question. How would I compute $Q^\pi(s,a)$ using the method above? My confusion comes from the fact that $\pi$ may assign different actions to $s$, with non-zero probability for actions $a'\neq a$ as well. So, whenever I sample, should I only apply $a$ when I am in state $s$, or should I only apply $a$ at $s$ at the very start, and then apply action at $s$ according to $\pi$? My guess is that the latter is the way to do. Is that right, and if it is - is it always the way I should think about $Q^\pi(s,a)$?
