Stochastic gradient descent for neural networks with tied weights

Question

For the neural network depicted below, I want to calculate the Error with respect to $w_{tied}$, which we get if we tie the weights $w_1$ and $w_4$ together. Tying the weights together would help to reduce overfitting, since we reduce the number of parameters. The hidden units $h_1,h_2$ are logistic, the output neuron f is a linear unit, and we are using the squared error cost function $E = (y − f)^2$.

I know that the solutions is (1) $\frac{\partial E}{\partial w_{tied}} = \frac{\partial E}{\partial f}(\frac{\partial f}{\partial h_1}\frac{\partial h_1}{\partial w_{1}}+\frac{\partial f}{\partial h_2}\frac{\partial h_2}{\partial w_{4}})$

From that it follows that

$\frac{\partial E}{\partial w_{tied}} = -2(y-f)(u_1*h_1*(1-h_1)*(-x_1)+ u_2*h_2*(1-h_2)*(-x_2))$

Question: How do I write the stochastic gradient descent algorithm for $w_{tied}$ now? In stochastic gradient descent, we randomly pick one datasample to optimize - Here, do we randomly pick a weight we want to optimize? So, we aren't we optimizing for the entire weight vector $\vec{w}$? But for random i we optimize $w_i$, by feeding all possible datasamples x into the gradient?

Also, I realize that in our solutions, it is "$x_1$" or "$x_2$" in the gradient above, and not "$-x_1$" or "$-x_2$" - why is that? Thanks

Answer 1

First of all, shouldn't your equation (1) be the following? $$ \frac{\partial E}{\partial w_{tied}} = \frac{\partial E}{\partial f}(\frac{\partial f}{\partial h_1}\frac{\partial h_1}{\partial w_1}+\frac{\partial f}{\partial h_2}\frac{\partial h_2}{\partial w_4}) $$ As the intuition is, since we are forcing $w_1$ and $w_4$ to be the same, to update them we should just take the optimal update for each and then average (or sum) them?

In a stochastic gradient descent step, we update all of the weights at once. So that doesn't change here, just for the weights in $w_{tied}$, their updates will both be the same, as per the equation above. All the other weights $w_i$ will be also updated during the step according to their normally-calculated gradients, $w_i = w_i - \text{[learning rate]} * \frac{\partial E}{\partial w_i}$.

For your last question, I get no $-$ sign in front of $x_1$. I think you just need to double-check your derivation / application of the chain rule.