# Why in LSTM cell the formula for the new hidden state applies activation function to the cell state without any transformation?

After reading this colah's blog about LSTM cells I wonder what would have happened if I changed the formula for the new hidden state $h_t$ a little bit.

Originally, using colah's notation we put

$$h_t = o_t * \tanh(C_t)$$

However, I would like to add a little complexity and add an affine transformation as in other "layers"

$$h_t = o_t * \tanh(W_h\cdot C_t+b_h)$$

Question Is there a good reason why to stick to the original formula?

I think to check heuristically if this tweak does any good to the models, but maybe I shouldn't mess with that.

## 1 Answer

You can try it, and maybe it will work better on some problems. The LSTM architecture is almost certainly not the optimal one -- it was just the first one to work.

For inspiration, also see this paper on trying out 10000 different LSTM-like architectures.

• I have already had a glance at Jozefowicz's paper, but I would rather see if there is some deeper concept behind the formula for $h_t$. – Fallen Apart Jul 17 '18 at 14:19
• @FallenApart I doubt it. – shimao Jul 17 '18 at 14:21
• I was inspired to ask this question due to the colah's statement "The cell state is kind of like a conveyor belt". Thus I thought that we somehow would like this conveyor belt to run into hidden state without any essential perturbations. But this is just my guess. – Fallen Apart Jul 17 '18 at 14:26
• @FallenApart but the $h_t$ you are asking about isn't conveying information along the cell state so much as it is pulling information from the conveyor belt. $C_t$ remains undisturbed here, no matter what sort of transformation you perform on it before computing $h_t$. – shimao Jul 17 '18 at 14:27
• You are right, I should have written "pulling information" instead of "run into". Still I wonder if there is any deeper reason behind the formula. :) – Fallen Apart Jul 17 '18 at 14:31