After reading this colah's blog about LSTM cells I wonder what would have happened if I changed the formula for the new hidden state $h_t$ a little bit.

Originally, using colah's notation we put

$$h_t = o_t * \tanh(C_t)$$

However, I would like to add a little complexity and add an affine transformation as in other "layers"

$$h_t = o_t * \tanh(W_h\cdot C_t+b_h)$$

Question Is there a good reason why to stick to the original formula?

I think to check heuristically if this tweak does any good to the models, but maybe I shouldn't mess with that.


You can try it, and maybe it will work better on some problems. The LSTM architecture is almost certainly not the optimal one -- it was just the first one to work.

For inspiration, also see this paper on trying out 10000 different LSTM-like architectures.

  • $\begingroup$ I have already had a glance at Jozefowicz's paper, but I would rather see if there is some deeper concept behind the formula for $h_t$. $\endgroup$ – Fallen Apart Jul 17 '18 at 14:19
  • $\begingroup$ @FallenApart I doubt it. $\endgroup$ – shimao Jul 17 '18 at 14:21
  • $\begingroup$ I was inspired to ask this question due to the colah's statement "The cell state is kind of like a conveyor belt". Thus I thought that we somehow would like this conveyor belt to run into hidden state without any essential perturbations. But this is just my guess. $\endgroup$ – Fallen Apart Jul 17 '18 at 14:26
  • $\begingroup$ @FallenApart but the $h_t$ you are asking about isn't conveying information along the cell state so much as it is pulling information from the conveyor belt. $C_t$ remains undisturbed here, no matter what sort of transformation you perform on it before computing $h_t$. $\endgroup$ – shimao Jul 17 '18 at 14:27
  • $\begingroup$ You are right, I should have written "pulling information" instead of "run into". Still I wonder if there is any deeper reason behind the formula. :) $\endgroup$ – Fallen Apart Jul 17 '18 at 14:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.