I'm training a reinforcement model playing a game with self-learning. For each state, the agent can select one or several simultaneous actions from a list of possible actions. One possible action is a binary choice that can make other actions irrelevant (not illegal, just irrelevant). Some actions do however become "illegal" for certain states. And some combinations of actions are illegal.
My plan is to produce a fixed-length vector of binary values indicating the agent's choice of actions given a state. I have two questions:
Question 1: In the tutorials I've done the algorithm typically selects ONE action from a list of possible actions using something like $\text{argmax}_{a \in > \mathcal{A}} \hat q(s,a)$. I want to instead predict an action vector, not just an action index. What is the best way to do this? The number of possible action vectors is huge, so I'm worried the task becomes very hard...
My second question is how to handle the legality of various combinations. Assuming there is a good way to predict a vector of actions per Question 1 (i.e. the agents act method outputs a vector [0,0,0,1,1,0,1] where each value indicates if action with index n is taken or not) I need to decide how to handle illegal combinations of actions.
The response to this question "How to handle a changing action space in Reinforcement Learning" claims that
You don't need to do anything special to handle [illegal actions]. The only thing you need to change is to not take any illegal actions. The typical Q-learning greedy policy is $\pi(s) = \text{argmax}_{a \in > \mathcal{A}} \hat q(s,a)$ and the epsilon-greedy rollout policy is very similar. Simply replace the action space $\mathcal{A}$ with just the legal actions $\mathcal{A}_\text{legal}(s)$.
Question 2: Am I correct to just run the prediction step as if all actions can be selected together, and then remove all vectors of illegal actions from the predictions before selecting which one to take? What if the network only produces illegal vectors? It feels like this might get very sparse...