1
$\begingroup$

This is a question that has been bugging me for some time: if I'm designing a RL algorithm to find the optimal value-action function Q(s,a), and I want the optimal action to depend on the specific state, shouldn't I always make sure that the function I use for Function Approximation (FA, be it linear, NN of other) includes the interaction between state variabes and the action variable?

If I do not include these interactions, each action would only contribute a fixed amount to the Q-value, so there would always be one optimal action, regardless of the state values, right?

When using a tabular approach (so, no FA) a different action may be optimal for each state. With linear FA, I can define a separate set of coefficients/weights for the state variables per action value (I have a limited number of action values). But how should I tackle this problem when using a NN for FA? Will the NN automatically/implicitly model the interactions (between actions and state-features) if I define enough hidden layers with non-linear activation functions? Any thoughts on which activation functions are most appropriate?

PS. I have the feeling I'm missing something, because I don't see much literature about interactions and Neural Nets. One exception is https://openreview.net/forum?id=ByOfBggRZ.

$\endgroup$
1
$\begingroup$

A Q-network generally takes in the state $s$ and produces $|A|$ outputs -- one for the Q-value of each action in $s$. In order to compute $\hat Q(s,a)$, you pass $s$ through the network, and then look at the appropriate output. Therefore, the network does implicitly model the "interaction".

$\endgroup$
  • $\begingroup$ OK, sound reasonable. But how is the network trained then? Each training sample requires |A| outputs, but after taking action a in state s we only have one output for that state? $\endgroup$ – dkd Aug 24 '18 at 7:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.