I had an attempt at training a convolutional LSTM network to imitate a player of the 2D platformer/shooter Teeworlds. This is an example of the data and labels I fed to the network:
https://www.youtube.com/watch?v=8ocxnWipQDw
The network was to predict the action of the player (on the right side in the video) based on a simplified image of what the player sees in the game (on the left). This is running at 25 frames per second and I collected 1.644.114 frames of data so far (collecting more shouldn't be an issue). The truncated backpropagation length is 300 frames, so the SGD algorithm works up to 12 seconds in the past, although state could be stored for longer in theory.
The Problem
For example one of the choices the player has is which weapon to select. But which weapon the player is holding is also indicated by the color of the player (always in the middle of the screen). The network just learns to look for which weapon the player is holding to output it as an action. I'm sure other actions the network can take will settle on this strategy as well.
So when the neural network is playing the game it will just stand still doing nothing.
I tried to mitigate this problem by offsetting data and labels by 1 frame in time. This matches reality where the action the player takes can only be visible in the next frame. Although this still gives no incentive for the network to switch weapons because it can achieve 99% accuracy with this strategy since players switch weapons rarely.
I could just omit the information about which weapon the player is holding in the image because the network has direct control over it anyways. But other actions such as walking right and left will still be a problem. I can't omit the information about the viewport moving right/left - when the player moves upwards he is likely to be pressing the jump button.
Are NNs the right tool?
I feel like there is a fundamental problem in the way I framed this problem to the neural network. Other "imitation" tasks like composing/writing in the style of Bach/Shakespeare never work with an environment the actions are based upon. They just feed previous actions (music notes/letters) to the network to predict the next one. But in a game the actions the player takes are highly dependent on the environment: You can be good at writing poems blindfolded, but not at a game with unpredictable enemies.
Are neural networks the wrong approach here? Are there papers or previous successes in similar areas? Is there a well known way to frame this kind of problem to a neural network?