4
$\begingroup$

I'm learning RNN and know that a basic RNN maps an input sequence to a sequence of hidden states, and then maps the sequence of hidden states to the output sequence. I'm confused about the dimension of the 3 sequences. The dimension of the input and output sequence can be found from data, but what about the dimension of the hidden states? Is it set by people in advance, or set to be the dimension of the output or input sequence?

$\endgroup$
3
  • 2
    $\begingroup$ You can fix it, but it's also a hyperparameter of the model. I'm working with an RNN model now; the model with 50 hidden states underfits, but the one with 400 overfits. $\endgroup$
    – Sycorax
    Commented Nov 1, 2016 at 23:07
  • 1
    $\begingroup$ This doesn't seem to be unanswerable to me. $\endgroup$ Commented Nov 2, 2016 at 1:31
  • $\begingroup$ I don't see what is primarily opinion based about this question. I'm voting to leave open. $\endgroup$ Commented May 27, 2017 at 11:48

1 Answer 1

6
$\begingroup$

The dimension of the RNN hidden state can be chosen in advance. It is independent of the dimension of the input or output sequences.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.