# Creating a neural network that can make a decision with optional arguments

I'm a final year computer science student and for my final year project I have to design a neural network to play a little known board game called 'The Downfall of Pompeii'.

I have to use reinforcement learning to train the neural network and I'll evaluate the value of states using temporal difference learning. This part doesn't confuse me much (yet!).

What is troubling me are the outputs of the network. With most states in the game the network will only be able to make 2/3 legal moves at a time but with these moves come several other optional arguments.

For example, in the game the player can draw a lava tile and place it on the board at several coordinates. Let's say the list of these possible coordinates were [A1, D7, E9]. Then, the moves the network could make are 'place A1', 'place D7' and 'place E9'. There are several other decisions that can be made in the game following this exact pattern i.e 'move X Y', where X is the coordinate we're moving from and Y is the coordinate we're moving to.

Of course, I'll have to feed the network with the list of possible coordinates it can choose but my question is this, how do I construct the outputs of the network so that it knows it has to choose a coordinate(s) from a list that it takes as an input. For example, an output will be 'move chosen-coordinate', but how will I get it to decide on that chosen coordinate from a list?

I'm quite new to neural nets but I have looked quite a bit before posting and the only thing I can find that may point me to a correct answer is Matthew Lai's paper: https://arxiv.org/abs/1509.01549. I'll be delving into this tomorrow but I was hoping I could find some guidance here too.

I'll be using Tensorflow/Keras to build the network. Although, I don't need any code examples if they can't be provided, merely direction on how I'd approach this.

If I haven't made anything clear please let me know!

## 1 Answer

Suppose you have three times of actions: A, which takes no arguments, B x which takes one argument x, and C x y which takes two arguments. Assume these arguments are chosen from a discrete candidate set (such as your list of coordinates).

Your network can output a categorical distribution $$S$$ across the 3 actions, a categorical distribution $$Q$$ across all possible arguments, and a categorical distribution $$R$$ across all ordered pairs of arguments.

To sample an action from this policy, first sample from $$S$$. If it's action A, you're done -- you can ignore all the other network outputs. If it's action B, then sample your argument x from distribution $$Q$$. If action C, then sample your two arguments x y from $$R$$.

Note that it's insufficient to use two categorical distributions across your candidate set instead of a distribution across ordered pairs of sets.

To cover some edge cases:

What if some of the coordinates in the candidate set are not valid for a particular action? Then keep sampling from your distribution until you get a valid coordinate.

What if you have a lot of coordinates and an action with a lot of arguments and it's not feasible to output a softmax across them all? Then you use an RNN to model the distribution of arguments.

What if the candidate set of arguments isn't fixed? (for example, suppose you must pick some turn t in the game -- obviously, it doesn't really make sense to have a softmax across turns in a game). But even this can be dealt with using Pointer Networks