I'm trying to understand how rnn's can be used to predict sequences by working through a simple example. Here is my simple network, consisting of one input, one hidden neuron, and one output:
The hidden neuron is the sigmoid function, and the output is taken to be a simple linear output. So, I think the network works as follows: if the hidden unit starts in state
s, and we are processing a data point that is a sequence of length $3$, $(x_1, x_2, x_3)$, then:
1, the predicted value, $p^1$, is
$$p^1 = u \times \sigma(ws+vx^1)$$
2, we have
$$p^2 = u \times \sigma\left(w \times \sigma(ws+vx^1)+vx^2\right)$$
3, we have
$$p^3 = u \times \sigma\left(w \times \sigma(w \times\sigma(ws+vx^1)+vx^2)+vx^3\right)$$
So far so good?
The "unrolled" rnn looks like this:
If we use a sum of square error term for the objective function, then how is it defined? On the whole sequence? In which case we would have something like $E=(p^1-x^1)^2+(p^2-x^2)^2+(p^3-x^3)^2$?
Are weights updated only once the entire sequence was looked at (in this case, the 3-point sequence)?
As for the the gradient with respect to the weights, we need to calculate $dE/dw, dE/dv, dE/du$, I will attempt to do simply by examining the 3 equations for $p^i$ above, if everything else looks correct. Besides doing it that way, this doesn't look like vanilla back-propagation to me, because the same parameters appear in different layers of the network. How do we adjust for that?
If anyone can help guide me through this toy example, I would be very appreciative.