First I want to check is I understand correctly how backprop works for NN with embed layers.
Lets create simplest NN with embed layer
- Input Layer [i1; i2; i3; i4] - input nodes
- Embed layer [h1] [h2] - here we have 2 nodes h1 and h2; h1 is connected to i1, i2 and h2 is connected to i3, i4. Corresponded weights of these connections w1, w2, w3, w4. Hence 2 embed blocks with only 1 node in each.
- Output layer with node y connected with outputs from embed layer. I.e two connections, h1 -> y, h2 -> y
Now I want to use back prop to calc gradient of weights.
As I understand shared weights between input layer and embed layer must always have the same values, i.e. w1 = w3, w2 = w4, so when initializing weights for network we should ensure that; also after each epoch these weights (shared) should remains equal, so after we calculate gradients for w1 and w3 (as normal) we should average them like this w1 = w3 = (w1 + w3) / 2 and w2 = w4 = (w2 + w4) / 2, this will ensure us shared weights will remains the same after each epoch.
Now if this is correct, then how could I use gradient check to test my backprop works, since gradient check knows nothing about restrictions of how we calculate gradients for shared weights in embed layer.