# Many-to-many or many-to-one LSTM when predicting a value derived from a sequence of features

Let's say I have a time series data set consisting of features that may correlate to whether or not the price of a stock will go up or down. Say these data points are at 5 minute intervals. I build an LSTM that takes in two hours of these sequential data points (24 time steps) and then attempts to predict if the price will have increased/decreased an hour after the last data point fed into the network.

As the training data is historic data, I have labels for every data point - whether or not the price increased/decreased an hour after this data point. In practice, I would be inputting a sequence of 24 data points, but will only base my prediction off of the final output. As the LSTM produces an output at each time step, should I be calculating the loss from all 24 of these outputs, or just the last one?.

The temporal nature of these data points is important for determining this up/down trend. Thus the output at the first time step will be a value derived from a single data point (with no previous sequence). I think that this would have a negative effect on the loss.

Or, do you think that because of the nature of an LSTM, and because it has learned on previous training sequences, it will actually make an accurate prediction on this single, first time step alone?

So can I include it, and every other time step's output, in the loss calculation?

$$\sigma(W x + b)$$ where $x = \begin{pmatrix} h_0 \\ h_1 \\ \vdots \\ h_{t-1}\end{pmatrix}$ is a vertically concatenated vector of outputs. $W\in \mathbb{R}^{24\times dim(h_i)\cdot t}$, $b\in\mathbb{R}^{24}$. Now you can use something like categorical crossentropy as your loss function.
• I am still unsure - With the notion of "unrolling" an RNN into the number of time steps in the inputs sequence (24 in my case), wouldn't each of these 24 time steps have a dense layer from the last LSTM layer to the output neurons? Say my last layer at each time step has a size of 64 LSTM cells and my network has two outputs at each time step (price is either higher/lower 1 hour from this time step), the weights corresponding to this dense layer would have a shape of $100 \times 2$. I use a softmax cross entropy on these two outputs - thus my LSTM produces a prediction at all 24 time steps.