This is a question that has been bugging me for some time: if I'm designing a RL algorithm to find the optimal value-action function Q(s,a), and I want the optimal action to depend on the specific state, shouldn't I always make sure that the function I use for Function Approximation (FA, be it linear, NN of other) includes the interaction between state variabes and the action variable?
If I do not include these interactions, each action would only contribute a fixed amount to the Q-value, so there would always be one optimal action, regardless of the state values, right?
When using a tabular approach (so, no FA) a different action may be optimal for each state. With linear FA, I can define a separate set of coefficients/weights for the state variables per action value (I have a limited number of action values). But how should I tackle this problem when using a NN for FA? Will the NN automatically/implicitly model the interactions (between actions and state-features) if I define enough hidden layers with non-linear activation functions? Any thoughts on which activation functions are most appropriate?
PS. I have the feeling I'm missing something, because I don't see much literature about interactions and Neural Nets. One exception is https://openreview.net/forum?id=ByOfBggRZ.