1
$\begingroup$

I want to train a recurrent policy gradient which predicts action probabilities based on prior environment states. However, I am unable to backpropagate during the "update policy" step, in which the running rewards are scaled, normalized, summed, and used to update the model. I understand that training an rnn in this context is unusual, because the RNN has to be manually unrolled, as the environment state depends on the last prediction in the sequence of predictions.

My unrolling scheme is:

  1. sample environment state
  2. state, hidden_in -> RNN cell -> hidden_out
  3. hidden_out -> linear function -> action probability
  4. hidden_out -> hidden_in

But I get this error during the backward step:

RuntimeError: grad can be implicitly created only for scalar outputs

Here is the code for the model and policy update:

import torch.nn as nn


class Policy(nn.Module):
    def __init__(self, state_space, action_space, hidden_size, n_layers, dropout_rate, gamma):
        super(Policy, self).__init__()
        self.input_size = state_space.shape[0]
        self.output_size = action_space.n
        self.hidden_size = hidden_size
        self.n_layers = n_layers

        self.rnn = nn.GRUCell(input_size=self.input_size,
                              hidden_size=self.hidden_size)

        self.relu = nn.LeakyReLU()
        self.linear = nn.Linear(self.hidden_size, self.output_size)
        self.softmax = nn.Softmax(dim=-1)

        self.dropout_rate = dropout_rate
        self.dropout = nn.Dropout(p=self.dropout_rate)
        self.gamma = gamma

        # history
        self.hidden_history = None
        self.policy_history = None
        self.reward_episode = None
        self.reward_episode_local = None

        self.reset_episode()

        # Overall reward and loss history
        self.reward_history = list()
        self.reward_history_local = list()
        self.loss_history = list()

    def reset_episode(self):
        # Episode policy and reward history
        self.hidden_history = list()
        self.policy_history = list()
        self.reward_episode = list()
        self.reward_episode_local = list()

    def forward(self, x):
        size = x.shape[0]
        x = x.view([1, size])   # batch size = 1

        if len(self.hidden_history) > 0:
            h_0 = self.hidden_history[-1]
        else:
            h_0 = None

        x = self.rnn(x, h_0)
        self.hidden_history.append(x)

        x = self.relu(x)
        x = self.linear(x)
        x = self.softmax(x)

        return x

...

def update_policy(policy, optimizer, e):
    R = 0
    rewards = []

    # Discount future rewards back to the present using gamma
    for r in reversed(policy.reward_episode):
        R = r + policy.gamma * R
        rewards.insert(0, R)

    # Scale rewards
    rewards = torch.FloatTensor(rewards)

    # Normalize rewards
    rewards = (rewards - rewards.mean()) / (rewards.std() + float(np.finfo(np.float32).eps))

    # Calculate loss
    policy_history = torch.stack(policy.policy_history)
    loss = (torch.mul(policy_history, rewards).mul(-1), -1)

    # Update network weights
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    # Save and intialize episode history counters
    policy.loss_history.append(loss.data[0])
    policy.reward_history.append(np.sum(policy.reward_episode))
    policy.reset_episode()

I am open to recommendations as to how to implement an RNN policy gradient, but primarily I would like to understand what is the cause of this error.

$\endgroup$
  • $\begingroup$ Hi Ryan, is your idea of RNN policy learning worthy? Did it perform well? I'm trying to implement the same idea right now. $\endgroup$ – Pavel Chernov Dec 11 '18 at 10:47
  • $\begingroup$ Hi Pavel, I found that it took much longer to train but it did remain stable for longer than the 1st order PG where each prediction only depends on the current state. I was running this on the cartpole problem, which I think is well suited for an RNN because it accrues momentum, and tends to fail when the cart accelerates too much to correct itself. I removed any upper duration limits on success before training. I can double check the results for you when I get home. I will also post the entire project including training and testing on github $\endgroup$ – Ryan Dec 11 '18 at 20:04
  • $\begingroup$ Thanks for information! I also noticed that you don't sum over loss tensor: loss = (torch.mul(policy_history, rewards).mul(-1), -1).sum() I'm not very familiar with PyTorch and don't know if it is right. But other implementations do sum over loss. $\endgroup$ – Pavel Chernov Dec 13 '18 at 8:44
  • $\begingroup$ sorry I totally forgot about this ! I will definitely get around to sharing the whole thing soon :) Yes you are correct that you could just sum the loss and then pytorch's autodifferentiation would calculate each node's gradient w.r.t the sum automatically. But I had to do this way because this is RL, and you need to pause the RNN's prediction after each output to send it to the environment, and then sample the environment to get the next input step for the RNN. This means you cant use Pytorch's simple nn.GRU(x) where x is your entire time series. You build your time series while predicting $\endgroup$ – Ryan Dec 13 '18 at 23:36
  • 1
    $\begingroup$ @PavelChernov So I finally got around to thoroughly testing and uploading my rnn PG, and it doesn't actualy appear to be better. I think I had not removed the training limits on the standard model. Both models will eventually converge on the maximum score, but it looks like the standard PG might be more stable. if you want run some demonstrations, just pull this and run "test_cart_pole.py" or "test_cart_pole_rnn.py": github.com/rlorigro/cartpole_policy_gradient $\endgroup$ – Ryan Dec 16 '18 at 0:51
1
$\begingroup$

Turns out this was a simple bug in which the shape of my calculated rewards (aka loss) did not match the shape of my output layer...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.