0
$\begingroup$

I am new to Reinforcement (Machine) Learning; I started my project with Q Learning (Tabular Q), which was easy to understand. I am now trying to write the code for Value Function Approximation.

I take my feature vector $f$, which is some instantaneous parameters from my simulation, of size $|x|$; my weight matrix $w$ is of size $|A|*|x|$. Thus $$Q(s,a;w)=w^Tf$$

I am following the algorithm given on the following link.

https://stats.stackexchange.com/questions/187110/how-to-fit-weights-into-q-values-with-linear-function-approximation#=

I have just a few little queries though, in the following equations

$$w^{t+1} = w^t + \alpha.L$$ $$L = [R^t + \gamma max_a Q(s^{t+1}, a; w^t) - Q(s^t, a^t; w^t)].\nabla_wQ(s,a;w)$$

$$\nabla_wQ(s,a;w) = f$$ where $t$ represents the time instant.

My questions are,

$(i)$ In expression for $L$, everything inside the square bracket is constant, right? $max_a Q(s^{t+1}, a; w^t)$ and $Q(s^t, a^t; w^t)$ are the maximum action value for next state, and current state action value, respectively?

$(ii)$ If brackets yield a constant, then L is a vector of size $|x|$. Now, how do we add that vector $L$ to matrix $w$? Is it like $w^{t}[A][:]+=L$, updating weights only at index $A$, the action taken at time instant $t$?

Thanks!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.