3
$\begingroup$

First I want to check is I understand correctly how backprop works for NN with embed layers.

Lets create simplest NN with embed layer

  1. Input Layer [i1; i2; i3; i4] - input nodes
  2. Embed layer [h1] [h2] - here we have 2 nodes h1 and h2; h1 is connected to i1, i2 and h2 is connected to i3, i4. Corresponded weights of these connections w1, w2, w3, w4. Hence 2 embed blocks with only 1 node in each.
  3. Output layer with node y connected with outputs from embed layer. I.e two connections, h1 -> y, h2 -> y

Now I want to use back prop to calc gradient of weights.

As I understand shared weights between input layer and embed layer must always have the same values, i.e. w1 = w3, w2 = w4, so when initializing weights for network we should ensure that; also after each epoch these weights (shared) should remains equal, so after we calculate gradients for w1 and w3 (as normal) we should average them like this w1 = w3 = (w1 + w3) / 2 and w2 = w4 = (w2 + w4) / 2, this will ensure us shared weights will remains the same after each epoch.

Now if this is correct, then how could I use gradient check to test my backprop works, since gradient check knows nothing about restrictions of how we calculate gradients for shared weights in embed layer.

Thanks.

$\endgroup$
1
$\begingroup$

First of, adding restrictions like that are a possibility and can work quite well in some case, however they certainly aren't mandatory. In the general case, they aren't there.
If you do want them, you usually calculate the gradients separately, average them and then add that average to both. That ensures that both weights remain the same

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.