Using Convolutional Neural Networks on Board Games

Question

I have troubles with my CNN. The code runs and my network learns something, but the performance is really poor. The goal is to play the game Connect 4. The network therefor receives a numpy array with the size (batch_size, row, cols, 2). The 2 stands for two channels. In channel one there are all places obtained by player one and on the other channels are all places obtained by player two.

It was up now for a few days but still doesn't manage to recognize horizontal lines. It knows four stones vertical is good, and it also stops the opponent from getting four vertically. But there is 0 awareness of playing four diagonally or horizontally. Do I maybe convolute over the wrong axis? This could be an explanation why it only recognizes those...

Here is the CNN part of my code:

    in_x = x = Input((self.observation_space[0], self.observation_space[1], 2))  # stack of own(6x7) and enemy(6x7) field

    x = Conv2D(128, 3, padding="same",
               kernel_regularizer=l2(1e-4))(x)
    x = BatchNormalization(axis=1)(x)
    x = Activation("relu")(x)

    for _ in range(2):
        x = self._build_residual_block(x)

    x = Conv2D(filters=1, kernel_size=1, kernel_regularizer=l2(1e-4))(x)
    x = BatchNormalization(axis=1)(x)
    x = Activation("relu")(x)
    x = Flatten()(x)
    policy_out = Dense(action_space, kernel_regularizer=l2(1e-4), activation="softmax", name="policy_out")(x)

    self.model = Model(in_x, policy_out, name="connect4_model")

    self.optimizer = SGD(lr=1e-2, momentum=0.9)
    self.model.compile(optimizer=self.optimizer, loss='mse')

def _build_residual_block(self, x):
    in_x = x
    x = Conv2D(filters=128, kernel_size=3, padding="same",
               kernel_regularizer=l2(1e-4))(x)
    x = BatchNormalization(axis=1)(x)
    x = Activation("relu")(x)
    x = Conv2D(filters=128, kernel_size=3, padding="same",
               kernel_regularizer=l2(1e-4))(x)
    x = BatchNormalization(axis=1)(x)
    x = Add()([in_x, x])
    x = Activation("relu")(x)
    return x

observation_space is (6,7), action space is (7)

EDIT 1: Just found one first huge mistake. I used the wrong loss. The author defined its own loss. I have used a MSE wich is definitely wrong. The author from the script I copied the network from has one interesting comment:

import keras.backend as K
...    
def objective_function_for_policy(y_true, y_pred):
        # can use categorical_crossentropy??
        return K.sum(-y_true * K.log(y_pred + K.epsilon()), axis=-1)

Is this actually completly the same as categorical_crossentropy? Looks like he wasn't sure either. The epsilon doesn't appear in the crossentropy I know.

Answer 1

OP points out in an edit that s/he was not using the same loss function as was used in the reference code. Because OP’s code didn’t match the reference, it’s not surprising that the RL agent found a different result.

Incidentally, this is why I recommend unit tests and careful debugging as a core part of building neural networks specifically and programming generally.

Answer 2

It's difficult to answer this without reproducible code. However, one thing that jumps out is you don't seem to be doing any pooling between convolutions. This might be making it difficult for your network to recognize longer patterns. So for example if you do a 3x3 convolution in the top left of the board, and a 3x3 on the top right, there doesn't seem to be any communication between the two, except through the dense layer. Consider adding a few pooling layers, or changing the padding="valid" in the last few convolutions to incur a contraction of spatial dimensions.

Answer 3

The problem was as discovered in EDIT 1 the use of MSE. Theoretically MSE might would work if trained for 1000 of hours, but cross-entropy performs much better. After training my network over night it's still a rather dumb player. However, it now has an awareness of horizontal and vertical lines. At least sometimes...

In my implementation only the policy is part of the network. In the implementation I copied the network from, the policy's values are calculated too, which makes it smarter in the end. I have to do some more fine tuning to get similar performance.