Given a single training example $x = (x_1, x_2, x_3)$ and output $y$, the goal is to write down the "sequence of calculations required to compute the squared error cost (called forward propagation)". The hidden units $h_1,h_2$ are logistic, the output neuron f is a linear unit, and we are using the squared error cost function $E = (y − f)^2$.
This is an exercise from an old exam (not homework, but self-study). The network looks as follows:
My first question is regarding the vocabulary: I am not sure what a linear unit is - is f(x) = x? Also, if I understand the exercise correctly, we only have to calculate the forward pass (?), but as an exercise for myself I am trying to derive the backward step as well.
Anyway, I would proceed as follows:
Forward pass:
For each $h_i$ we sum over the respective weights time inputs. The input $h_{1_{in}}$ to ${h_1}$ for instance is $w_1*x_1+w_3*x_2+w_5*x_3$.
We apply the sigmoid function to these inputs to get the outputs $h_{i_{out}}$ of the $h_i$.
The weighted sum $u_1*h_{1_{out}}+u_2*h_{2_{out}}$ is then the input $f_{in}$ to the output node f.
Backward pass
Let's take as an example the weight $u_1$
$\frac{\partial E}{\partial u_1} = \frac{\partial E}{\partial f_{out}}*\frac{\partial f_{out}}{\partial f_{in}}*\frac{\partial f_{in}}{\partial u_1}$
Step by step I would calculate:
$\frac{\partial E}{\partial f_{out}} = -2*(y-f_{out})$
$\frac{\partial f_{out}}{\partial f_{in}}$; Assuming here f(x) = x, $f_{out} = f_{in}$. So I would have to derive $h_{1_{out}}*u_1+h_{2_{out}}*u_2$ with respect to itself and would end up with 1 ....?!
$\frac{\partial f_{in}}{\partial u_1} = h_{1_{out}} $
Taken together $\frac{\partial E}{\partial u_1} = -2*(y-f_{out})*1*h_{1_{out}}$ - to then update $u_1 = u_1 - \eta * \frac{\partial E}{\partial u_1}$
Is this correct? Thanks