0
$\begingroup$

I was looking at Open Ai's actor-critic code and noticed that some of the neural network's weights are shared

class Policy(nn.Module):
    def __init__(self):
        super(Policy, self).__init__()
        self.affine1 = nn.Linear(4, 128)
        self.action_head = nn.Linear(128, 2)
        self.value_head = nn.Linear(128, 1)

        self.saved_actions = []
        self.rewards = []

     def forward(self, x):
        x = F.relu(self.affine1(x))
        action_scores = self.action_head(x)
        state_values = self.value_head(x)
        return F.softmax(action_scores, dim=-1), state_values

namely affine1.

I am not sure why this is possible, because, theoretically,

  • The actor should calculate a function from states to actions

  • The critic should calculate a function from states to reward

These are two different functions, thus I would expect separate weights.

The only reason I came up with was a projection of exterior information on the problem, which is "Embedding the state".

If that is the case, then who is to say it makes sense to use he same state embedding for both networks?

Why does weight sharing in actor critic make sense?

$\endgroup$
2
$\begingroup$

You can use weight-sharing almost any time you might want to make more than one prediction given some input. The shared layer(s) construct some representation which is useful for all the prediction tasks. This is effective because the loss from all down stream prediction tasks backpropagates to this layer, allowing faster and more efficient learning.

These are two different functions, thus I would expect separate weights.

The last layers "head" have different weights and thus compute different outputs.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.