For the neural network depicted below, I want to calculate the Error with respect to $w_{tied}$, which we get if we tie the weights $w_1$ and $w_4$ together. Tying the weights together would help to reduce overfitting, since we reduce the number of parameters. The hidden units $h_1,h_2$ are logistic, the output neuron f is a linear unit, and we are using the squared error cost function $E = (y − f)^2$.

I know that the solutions is (1) $\frac{\partial E}{\partial w_{tied}} = \frac{\partial E}{\partial f}(\frac{\partial f}{\partial h_1}\frac{\partial h_1}{\partial w_{1}}+\frac{\partial f}{\partial h_2}\frac{\partial h_2}{\partial w_{4}})$

From that it follows that

$\frac{\partial E}{\partial w_{tied}} = -2(y-f)(u_1*h_1*(1-h_1)*(-x_1)+ u_2*h_2*(1-h_2)*(-x_2))$

Question: How do I write the stochastic gradient descent algorithm for $w_{tied}$ now? In stochastic gradient descent, we randomly pick one datasample to optimize - Here, do we randomly pick a weight we want to optimize? So, we aren't we optimizing for the entire weight vector $\vec{w}$? But for random i we optimize $w_i$, by feeding all possible datasamples x into the gradient?

Also, I realize that in our solutions, it is "$x_1$" or "$x_2$" in the gradient above, and not "$-x_1$" or "$-x_2$" - why is that? Thanks

enter image description here


1 Answer 1


First of all, shouldn't your equation (1) be the following? $$ \frac{\partial E}{\partial w_{tied}} = \frac{\partial E}{\partial f}(\frac{\partial f}{\partial h_1}\frac{\partial h_1}{\partial w_1}+\frac{\partial f}{\partial h_2}\frac{\partial h_2}{\partial w_4}) $$ As the intuition is, since we are forcing $w_1$ and $w_4$ to be the same, to update them we should just take the optimal update for each and then average (or sum) them?

In a stochastic gradient descent step, we update all of the weights at once. So that doesn't change here, just for the weights in $w_{tied}$, their updates will both be the same, as per the equation above. All the other weights $w_i$ will be also updated during the step according to their normally-calculated gradients, $w_i = w_i - \text{[learning rate]} * \frac{\partial E}{\partial w_i}$.

For your last question, I get no $-$ sign in front of $x_1$. I think you just need to double-check your derivation / application of the chain rule.

  • $\begingroup$ I corrected the notation, you are right. However, what is the stochastic part here? The point in SGD is that we do pick datapoints at random..so how does that translate here? $\endgroup$
    – Pugl
    Commented Aug 8, 2017 at 19:17
  • 1
    $\begingroup$ The stochastic part is that you will pick data points at random. Unless you only have one [$x_1, x_2, x_3$] vector, then you'll just 'stochastically' pick that one over and over again. Have never heard of stochastically updating weights (although it could be useful it would probably just be inefficient) - all weights are always updated at every training step. $\endgroup$
    – nlml
    Commented Aug 8, 2017 at 19:32
  • $\begingroup$ actually realized that it was correct before I edited it, at least in our solutions it is w_tied...hm, well.. $\endgroup$
    – Pugl
    Commented Aug 9, 2017 at 21:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.