I am referring to page 199 of Sutton and Barto book on Reinforcement Learning available here: book

There the Mean Squared Value Error for an vector-parameterized function approximation $\hat{v}(s,\mathbf{w})$ of the true value function of a given policy $\pi$ is defined as

$$\overline{VE}(\mathbf{w}) \equiv \sum_{s \in \mathcal{S}} \mu (s)\left[v_{\pi}(s)- \hat{v}(s,\mathbf{w})\right]^2$$

where $\mu(s)$ is some distribution ($\mu(s) \geq 0, \sum_s \mu(s) =1$) over $\mathcal{S}$.

What would be its definition with respect to action-value approximation functions $\hat{q}(s,a,\mathbf{w})$? Is it correct, by recalling that $v_{\pi}(s)= \sum_a \pi(a|s)q_{\pi}(s,a)$, to define it as

$$\overline{VE}(\mathbf{w}) \equiv \sum_{s \in \mathcal{S}} \mu (s) \sum_{a \in \mathcal{A}(s)} \pi(a|s)\left[q_{\pi}(s,a)- \hat{q}(s,a,\mathbf{w})\right]^2$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.