Multi stage neural net

Question

I'm going to use my specific use case here, but I'm curious about this AI structure in general.

I want to create an AI to play Rocket League (a video game). For context, in Rocket League you control a car which can drive around, jump, and "boost" (kind of fly thru the air like a ""rocket""). In the 2v2 format, yourself and a teammate attempt to score a ball into the enemy net by hitting it with your cars, while the enemies do the same. Think soccer with cars.

At the final stage of my algorithm, I have two neural networks. First, I have a recurrent neural network, network A that should have:

• Input: position and rotation of all objects in the field (players and ball) for the past 15 seconds (time is arbitrary, tuned later)
• Output: wanted position, rotation, velocity, and angular velocity of my car. I'll abbreviate this set on info P.

The second network, network B, will likely be a simple feed forward net. It should have:

• Input: the wanted P and the current P
• Output: keyboard inputs. For example, it might output that the player should press the turn left button and the jump button.

In order to train the networks, I thought it best to train network B first. I have saved replays of my car's position throughout a game, and soon I will have associated keyboard inputs saved as well, so I should have training data easily.

To train B, I would choose a set T of times throughout my replay (maybe roughly every 5 seconds). Then, for each each T(n), and T(n-1) < t < T(n), my training data would be:

• Input: wanted P = P(T(n)), current P = P(t)
• Output: keyboard inputs at t

Once this network is training, my AI should be able to move it the car to any wanted P. Now, to train network A, I would use:

• Input: Past 15 seconds of P data from the replay I mentioned above
• Output: keyboard inputs

The bit here that I'm worried about is: I'm training network A kind of "through" network B. Once B has been trained, its weights/biases shouldn't be modified. So will I be able to backprop succesfully? I'd basically be skipping over all the the layers in network B.

I'm also interested in this output of this neural net, because it will act like one large network, but the layer in the middle which divides A from B will actually have meaningful values. In theory this means A will be like the decision making part, and B will be like the mechanics part.

Is this a good idea? Am I trying to solve a problem which already has been solved? Is there an established better approach to this?

Answer 1

So will I be able to backprop succesfully?

Yes, computing derivatives through backpropagation works just fine. The composition of two differentiable functions (in this case, neural networks) gives another differentiable function, so whatever deep learning framework you are using, its auto-differentiation should handle these computations. In PyTorch, your training loop for A will look like once you have already trained B.

criterion = MyCustomCriterion() # e.g. CrossEntropy, MSE, etc.
netA = NetA() # instance of some subclass of torch.nn.Module
netB = NetB(pretrained=True) # instance of some subclass of torch.nn.Module
optimizerA = torch.optim.Adam(params=netA.parameters())
for epoch in range(num_epochs):
    for batch, label in dataloader:
        netA.zero_grad()
        netB.zero_grad()
        outputA = netB(batch)
        outputB = netA(outputA)
        err = criterion(outputB, label)
        # Calculate gradients for all parameters in backward pass.
        # Keep in mind that computing gradients with respect to the parameters of A
        # requires computing the gradients with respect to B through the chain rule.
        err.backward()
        optimizerA.step()

I think this is a very normal approach to this since you are interested in using the values for the intermediate layers. Otherwise, it might be easier (and possibly get better results) if you just used one larger neural network.