How to properly develop a machine learning model for a poker game?

I've created an annotation for poker games, similar to chess games. After compiling information from thousands of games, I want to use this large data set for machine learning.

To simplify, let's focus on the last betting round of a Texas Hold'em game. The five cards in the table are visible, and the computer needs to take a decision (fold, check, bet or raise). Of course, each action is dependent on the circumstances of the game (e.g. you can only fold if someone has placed a bet).

So, the decision is somewhat between a classification (fold, check, bet) and a regression (bet \$1,000, bet \$1,500, bet \$2,125, etc). And the result should not be 'correct' or 'incorrect', but a payout over your decision. For instance, if you have bet \$1,500 and the others have folded, then you "win" \$1,500. But if someone has a better game than you, then you lost -1 * your bet, or minus \$1,500.

I need to reinforce that there is not a correct option. In poker, you need to maximize your returns in the long run. The exact same decision in an exact similar situation can have different outcomes if the opponent shows a pair of aces or a pair of threes in the last round -- and the computer can't know when to fold and when to bet in this situation, it needs to be consistent in the long run.

So, let's include some requisites:

• The overall technique 'result' should be the sum of all the individual instance results;
• It would be good to have conditional decisions, e.g. you check, the opponent raises, and then you need to decide again based on this action;
• The main goal of this modeling is to implement different machine learning techniques, and use the data set to evaluate them.

Any suggestions of an existing technique which I should base my modeling?

• This would appear to be a combination of two problems: modeling the utility of alternative game actions and probability models for those poker hands. The probability of poker hands and outcomes is well-covered already, but you'll need to make your own models for what the utility of game actions is. Some gamblers are risk takers, some are not, but it's not necessarily obvious which is better. What's optimal in this sense depends on specific goals. Moreover, I'm not sure that machine learning is particularly relevant here, so much as a utility model.
– Sycorax
Sep 23, 2015 at 4:50
• I think it would be wise to label the players and try to cluster them, i.e. are they risk takers or conservative. As @user777 explains, all the probabilities of poker hands are known. It is maybe better/safer to start of as a conservative player and try to figure out what kind of players you are against and try to add that into your model. There are also plenty of other features that you can add, such as pot odds and other things. Sep 23, 2015 at 7:43
• I would start with some basic EV. Get some reads on how often players will bluff and fold. Check out Sklansky Theory of Poker. ML with poker is very advanced. Mar 21, 2016 at 15:05
• Your modelling approach/assumptions appear to disregard betting and positioning information. It seems to focus almost exclusively to card equity. Yeah, this isn't going to be an straight-forward ride for you. Also just google "Poker Bots" and see the Wikipedia article on Computer Poker Players. You seem to (unintentionally) turn a blind eye to a lot of specialised poker literature. (+1 to @Paparazzi) Jan 6, 2017 at 11:11

1 Answer

Generally, reinforcement learning is what you're looking for, though most reinforcement learning implementations focus on perfect information games (e.g. pong), not imperfect information games (e.g. poker).

A deep reinforcement learning algorithm has recently achieved super-human play in full-scale Limit Hold'em. Of the machine learning approaches to AI poker, this is state-of-the-art, though it is still ~5BB/100 hands worse than the best domain-specific expert AI (excluding the AI that effectively solved the game).

I suspect we'll see deep reinforcement learning approaches to no-limit start to be competitive with top domain-specific expert AI in the next 5 years.