New answers tagged

2

Your input-to-hidden matrix $W_{hx}$ has shape $M \times K$. Your hidden matrix $W_{hh}$ has shape $M \times M$. Then $h_t, b_h, a_t$ all have shape $M$. The output matrix $W_{yh}$ has shape $K \times M$, so $W_{yh}h_t$ has shape $K$. Softmax doesn't change any shapes, so your output is $K$. You seemed confused about whether to think about $x_t, h_t$, etc as ...


5

Yes, RNNs can work with arbitrary and variable sized inputs, and backpropagation can be done with these arbitrary and variable sized computation graphs. You still have to "unroll" the graph (and have enough memory to do so) -- in some sense, there is no such thing as a "rolled graph", that's just a nice abstraction for humans to look at, ...


2

I believe you're right. The paper seems to use the numerator layout as the chain rule expands rightwards, which is the same as your calculations. So, for example, if the equation was $x=Wz$, and we were interested in $\frac{\partial Wz}{\partial z}$, the answer would have been $W$, not $W^T$. You can look at the third entry here. This simple example assumes $...


Top 50 recent answers are included