6
$\begingroup$

In autoencoders when using tied weights, it was mentioned the gradient with respect to w is the sum of two weights. I didn't understand this, can someone elucidate this.

it is mentioned here on slide 4 in the end.

$\endgroup$
0

1 Answer 1

5
$\begingroup$

Following the notation in the slides, a one layer autoencoder with tied weights is given by $$o(\hat{a}(x))=o(c+W^Th(x))=o(c+W^T\sigma(b+Wx))$$

The gradient wrt $W$ according to the product rule $$\frac{\partial l}{\partial W_{ij}}=\frac{\partial l}{\partial \hat{a}_j}\frac{\partial \hat{a}_j}{\partial W_{ij}}=\frac{\partial l}{\partial \hat{a}_j}(h_i+W_{ij}\frac{\partial h_i}{\partial W_{ij}})=\frac{\partial l}{\partial \hat{a}_j}h_i+\frac{\partial l}{\partial \hat{a}_i}x_j$$ which is equal to adding up the backpropagation gradients of each layer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.