What is the difference in ANN with lagged observation as predictors and LSTM? I am trying to understand how ANN is different from LSTM when we include lagged variables as predictors. Is the difference solely in forget gate for LSTM?
 A: LSTMs inputs include "memory" in the form of the hidden and cell states, in addition to the raw features. ANNs with lagged observations only have lags of the raw features, but not memory.
A: TLDR; LSTM is a recurrent (or autoregressive) model that uses it's own output as input in later time steps. An ANN with lagged variables as inputs isn't autoregressive at all since it's inputs are all exogenous.
Maybe historical development will put it into perspective. The recurrent neural network (RNN) was developed to allow for the OUTPUT of the ANN to be used as input at the next time step. So the ANN with lagged INPUTS differs from an RNN in the sense that the latter uses it's own OUTPUT in a previous step as input, instead of (or in addition to) lagged INTPUTS.
We know from econometric time series modeling (e.g. AR(p) models) that the problem with using your own output as input will cause your predictions to explode over time if you have a coefficient that in absolute value is larger than one. To remedy this we have to impose the restriction that the autoregressive coefficient (i.e. the one that relates to the models' own output in a previous time-step) is smaller than one. This restriction is imposed for AR models but for the RNN as well.
The problem with this restriction for the RNN, however, is what's called the vanishing gradient problem. Due to the requirement that contribution of earlier OUTPUTS die out as time progresses, the model isn't able to handle long term relations in the data.
Finally then, the LSTM is a model that uses it's own OUTPUT at time t as INPUT for time t+1 but remedies the vanishing gradient problem by allowing the gradient to flow unchanged.
