Are neural networks the wrong tool to solve this 2D platformer/shooter game? Is there a proven way to frame this problem to a neural network?

Question

I had an attempt at training a convolutional LSTM network to imitate a player of the 2D platformer/shooter Teeworlds. This is an example of the data and labels I fed to the network:

https://www.youtube.com/watch?v=8ocxnWipQDw

The network was to predict the action of the player (on the right side in the video) based on a simplified image of what the player sees in the game (on the left). This is running at 25 frames per second and I collected 1.644.114 frames of data so far (collecting more shouldn't be an issue). The truncated backpropagation length is 300 frames, so the SGD algorithm works up to 12 seconds in the past, although state could be stored for longer in theory.

The Problem

For example one of the choices the player has is which weapon to select. But which weapon the player is holding is also indicated by the color of the player (always in the middle of the screen). The network just learns to look for which weapon the player is holding to output it as an action. I'm sure other actions the network can take will settle on this strategy as well.

So when the neural network is playing the game it will just stand still doing nothing.

I tried to mitigate this problem by offsetting data and labels by 1 frame in time. This matches reality where the action the player takes can only be visible in the next frame. Although this still gives no incentive for the network to switch weapons because it can achieve 99% accuracy with this strategy since players switch weapons rarely.

I could just omit the information about which weapon the player is holding in the image because the network has direct control over it anyways. But other actions such as walking right and left will still be a problem. I can't omit the information about the viewport moving right/left - when the player moves upwards he is likely to be pressing the jump button.

Are NNs the right tool?

I feel like there is a fundamental problem in the way I framed this problem to the neural network. Other "imitation" tasks like composing/writing in the style of Bach/Shakespeare never work with an environment the actions are based upon. They just feed previous actions (music notes/letters) to the network to predict the next one. But in a game the actions the player takes are highly dependent on the environment: You can be good at writing poems blindfolded, but not at a game with unpredictable enemies.

Are neural networks the wrong approach here? Are there papers or previous successes in similar areas? Is there a well known way to frame this kind of problem to a neural network?

Answer 1

Yes, neural networks can learn how to play video games.

reinforcement-learning (RL) is the standard approach to solving game-playing using neural networks. A key paper in this area is Deepmind's Atari-playing RL agent, but researchers have extended this approach to more complex games like Doom, Starcraft II and DOTA. If you're not familiar with this research, I'd suggest picking up Maxim Lapan's Deep Reinforcement Learning Hands-On. The core idea of reinforcement learning is that the agent learns how to interpret the current state of the environment and take actions to maximize its payoff. RL is a different strategy compared to language-generating models, because it's not merely imitating past results; instead, it's learning how to make decisions to maximize rewards, even when the environment is changing.

That said, it's possible that the strategy you're using is not the best approach to solving this problem. Neural network construction involves a fair amount of experimentation to get a result that is good enough for your particular needs; for elaboration on this, see What should I do when my neural network doesn't learn?

Second, a key exception to Deepmind's success is Metroid-like games such as Montezuma's Revenge, which are harder for the agent to learn because there are extremely long lags between actions (picking up torches or gems) and their payoffs. If your platform game has a similar kind of lag between action and payoff, then your model may struggle.

And, as always, you should write unit tests and validate that your code is doing what you want it to.

My suggestion is to start with a problem that you know can be solved to a certain level of success by neural networks, such as Breakout, and see how well your method works. If your method can't learn to play Breakout, it might not be a viable strategy.