How would you structure the observation for a poker game

Question

After reading yet another paper about a Texas holdem AI, I am motivated to try out reinforcement learning methods instead of the Monte Carlo methods used in all the recent papers. (Since no paper actually used RL I don't expect this to be a very successful experiment, also I simplified the problem for now: Players can only raise by a fixed size of 1 Big blind, more bet sizes to come) But there's one problem: I am not sure how to structure the observation vector which I feed into PPO - I am thinking about putting the data in a 1D array and scaling the values so they are in a 0-1 range The question is: how do I encode what happened in the game into a vector? Here are some attributes that definitely have to be part of the observation (each card is two fields: rank and suit):

• Player hand
• Player stack size
• Player position relative to button
• Pot size (relative to stack size?)
• Amount to call
• Big blind size (because that is currently the only raise that is supported)
• for each opponent: Last action (fold, call, raise - might be from this betting round if the player sits behind opponent and from the last one if opponent sits behind player), Stack size, hand (if there is a showdown, before and otherwise -1 if the cards are unknown?), Position relative to button

There are probably a few other observations I could add, but my question is: Is it even possible that this could work? The opponents' card fields are almost always going to be -1, except in the rare cases where a showdown occurs, also I have to train a different network for each number of players at the table, which just screams "there's gotta be a better way" and prohibits me from teaching heads up (two player) poker first and then using more players, which at least reduces the computation time per iteration. Also: Which neural network types would you propose for a problem such as this? Since I want to apply proximal policy optimization, most of the network structure is clear, but would you try recurrent or convolutional input layers (and restructure the input data) or just use dense/fully connected layers all the way through?

Thanks in advance! Taxel

Answer 1

I actually used Reinforcement Learning to build poker AI's for my thesis; http://doras.dcu.ie/16765/ .

The most interesting aspect of building a poker AI is how to include previous actions; there is a lot of room for creativity. I used belief vectors from a Hidden Markov Model but I'm not sure that's the best way. I like your idea of including the last action of each player (I would include my own last action as well); that might be the simplest way of including previous actions. If that works you could include the last two actions etc..

To cope with multiple opponents, I added together vectors for each opponent. Again, there is a lot of room for creativity. Note: when decent players play most versions of poker, most hands get down to two player very quickly; usually pre-flop or flop.

My main recommendation would be to get to a system that takes legal actions in all situations as quickly as possible. And then experiment with the elements that most interest you. Have fun!