I want to implement a Recurrent Neural Newtork (RNN) and use it for a classification task. I can handle a Feed Forward Neural Network and I followed this blog tutorial to learn more about the implementation of a RNN. My main concern is about the dimension of $x_t$ because the tutorial encodes words in one-hot vectors, which I don't need. Consider the computation:

$$h_t = \sigma(Ux_t + Wh_{t-1})$$

Here $\sigma$ is some activation function and $h_t$ refers to the "hidden" state in $t$. Given that one input example $x$ would be a sequence of real valued numbers, say 784 numbers if I use the MNIST dataset for testing, and such a sequence belongs to one class, what would be the dimension of $x_t$? I suspect it is just a scalar (1 by 1, esentially one number of the sequence) but I would like to confirm with someone who has experience with RNNs. The dimension of matrix $U$ would be HiddenLayerSize by DimOfX_t and matrix $W$ would be of dimension HiddenLayerSize by HiddenLayerSize, I believe. What would need to adapted for a regression task? Many thx for any help in advance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.