# Modeling interaction between state and action in Reinforcement Learning with Function Approximation

This is a question that has been bugging me for some time: if I'm designing a RL algorithm to find the optimal value-action function Q(s,a), and I want the optimal action to depend on the specific state, shouldn't I always make sure that the function I use for Function Approximation (FA, be it linear, NN of other) includes the interaction between state variabes and the action variable?

If I do not include these interactions, each action would only contribute a fixed amount to the Q-value, so there would always be one optimal action, regardless of the state values, right?

When using a tabular approach (so, no FA) a different action may be optimal for each state. With linear FA, I can define a separate set of coefficients/weights for the state variables per action value (I have a limited number of action values). But how should I tackle this problem when using a NN for FA? Will the NN automatically/implicitly model the interactions (between actions and state-features) if I define enough hidden layers with non-linear activation functions? Any thoughts on which activation functions are most appropriate?

PS. I have the feeling I'm missing something, because I don't see much literature about interactions and Neural Nets. One exception is https://openreview.net/forum?id=ByOfBggRZ.

A Q-network generally takes in the state $s$ and produces $|A|$ outputs -- one for the Q-value of each action in $s$. In order to compute $\hat Q(s,a)$, you pass $s$ through the network, and then look at the appropriate output. Therefore, the network does implicitly model the "interaction".