After reading yet another paper about a Texas holdem AI, I am motivated to try out reinforcement learning methods instead of the Monte Carlo methods used in all the recent papers. (Since no paper actually used RL I don't expect this to be a very successful experiment, also I simplified the problem for now: Players can only raise by a fixed size of 1 Big blind, more bet sizes to come) But there's one problem: I am not sure how to structure the observation vector which I feed into PPO - I am thinking about putting the data in a 1D array and scaling the values so they are in a 0-1 range The question is: how do I encode what happened in the game into a vector? Here are some attributes that definitely have to be part of the observation (each card is two fields: rank and suit):
- Player hand
- Player stack size
- Player position relative to button
- Pot size (relative to stack size?)
- Amount to call
- Big blind size (because that is currently the only raise that is supported)
- for each opponent: Last action (fold, call, raise - might be from this betting round if the player sits behind opponent and from the last one if opponent sits behind player), Stack size, hand (if there is a showdown, before and otherwise -1 if the cards are unknown?), Position relative to button
There are probably a few other observations I could add, but my question is: Is it even possible that this could work? The opponents' card fields are almost always going to be -1, except in the rare cases where a showdown occurs, also I have to train a different network for each number of players at the table, which just screams "there's gotta be a better way" and prohibits me from teaching heads up (two player) poker first and then using more players, which at least reduces the computation time per iteration. Also: Which neural network types would you propose for a problem such as this? Since I want to apply proximal policy optimization, most of the network structure is clear, but would you try recurrent or convolutional input layers (and restructure the input data) or just use dense/fully connected layers all the way through?
Thanks in advance! Taxel