# Reinforcement Learning - Value Function Approximation

I am new to Reinforcement (Machine) Learning; I started my project with Q Learning (Tabular Q), which was easy to understand. I am now trying to write the code for Value Function Approximation.

I take my feature vector $$f$$, which is some instantaneous parameters from my simulation, of size $$|x|$$; my weight matrix $$w$$ is of size $$|A|*|x|$$. Thus $$Q(s,a;w)=w^Tf$$

I am following the algorithm given on the following link.

https://stats.stackexchange.com/questions/187110/how-to-fit-weights-into-q-values-with-linear-function-approximation#=

I have just a few little queries though, in the following equations

$$w^{t+1} = w^t + \alpha.L$$ $$L = [R^t + \gamma max_a Q(s^{t+1}, a; w^t) - Q(s^t, a^t; w^t)].\nabla_wQ(s,a;w)$$

$$\nabla_wQ(s,a;w) = f$$ where $$t$$ represents the time instant.

My questions are,

$$(i)$$ In expression for $$L$$, everything inside the square bracket is constant, right? $$max_a Q(s^{t+1}, a; w^t)$$ and $$Q(s^t, a^t; w^t)$$ are the maximum action value for next state, and current state action value, respectively?

$$(ii)$$ If brackets yield a constant, then L is a vector of size $$|x|$$. Now, how do we add that vector $$L$$ to matrix $$w$$? Is it like $$w^{t}[A][:]+=L$$, updating weights only at index $$A$$, the action taken at time instant $$t$$?

Thanks!