# Predicting sequence of integers / binary values

The data I am working with are sequences of binary values (0 / 1) which generally have a pattern - a longer sequence of 1's followed by a shorter sequence of 0's, for instance:
1 1 1 1 0 0 1 1 1 1 1 0 0 0
The goal is to predict the next value based on the entire sequence.
Each sequence has 10 to 20 values in it and the prediction should be made for each sequence individually by using all its available data. I realize there are few data points to work with so my question is: are any statistical methods than could achieve this type of prediction? I've never had to work with sequence data, so I don't know how to approach this.

• What is your goal here? Do you have a series of historical values and want to predict a few steps into the future based on that? Or do you want to model that time series so that you can artificially produce timeseries that behave in a similar way (to run simulation on etc.)? Commented Aug 25, 2017 at 13:21
• @user7019377 : Added a new line to clarify that I'd like to predict the next value in the sequence and that it should be predicted based on all the values in the sequence up to that point. Commented Aug 25, 2017 at 14:03
• Are these completely unknown signals from outer space.... Or, is there any possibility to have additional 'information' about the (currently abstract) time sequences? All pieces of additional information, not just the raw variables, can change the probability of the next 0 or 1 and improve the success rate of the predictions. Commented Aug 25, 2017 at 14:38
• @MartijnWeterings - all additional information available is dependent on the 0/1 values; the data shows behavior for real people, but the reasons for them performing (1) or not performing (0) a certain action cannot be known through the data. Commented Aug 29, 2017 at 9:57
• BogdanC, can you explain a bit more? Quantitative analysis relies on assumptions. There is much more that can be done if it is known what those 1's and 0's are, where they come from, what they mean, how they are produced, what the order in the sequence means, what relations are allowed to exist between different numbers in a single sequence and between different sequences. Presenting some numbers, without meaning, and asking consult for statistical methods is a minimal way to make something out of it and only gives you general solutions and advice. Commented Aug 30, 2017 at 10:17

One approach you could consider is trying to learn a Markov Chain (MC) to represent each sequence and then predict future values based on this MC.

MCs are a way of representing types of learning automata (LA) and can be used when the subsequent state of a system depends solely on the current state. They can be intuitively represented diagrammatically:

This is a very simple LA. It has two states: one where the last number seen was a 1 and one where the last number seen was a 0. There are transition probabilities between the different states noted as well. For example, when the LA is in state 0 it will stay in state 0 with probability $x$ and will move to state 1 with probability $1-x$. This can also be shown in the form of a matrix:

$\begin{bmatrix}x & 1-x \\ 1-y & y\end{bmatrix}$

Estimating from your example sequence, $1 1 1 1 0 0 1 1 1 1 1 0 0 0$, we might say that in this case $x = 0.6$ and $y = 0.77$.

This kind of solution can also be extended; we could learn an LA with more states and more "memory."

or

$\begin{bmatrix}w & 0 & 1-w & 0 \\ x & 0 & 1-x & 0 \\ 0 & 1-y & 0 & y \\ 0 & 1-z & 0 & z\end{bmatrix}$

This LA has four states: 00, where two or more consecutive 0s have been seen; 0, where only one consecutive 0 has been seen; 1, where only one consecutive 1 has been seen; and 11, where two or more consecutive 1s have been seen.

We can again estimate the corresponding probabilities from your example sequence and might say that $w = 0.33$, $x = 1$, $y = 1$ and $z = 0.71$.

• could you show how to the matrix transformation (squared of 2 dimensions, x, y) is applied to the vector in input 11110011111000 ? Commented Jun 12, 2021 at 8:36

One way to do it is through classification. You need a binary output, which is exactly what classification algorithms can provide.

You can construct a data-set out of this time series. Assume you have $n$ values right now. Further assume that you think the value in $n+1$ can be determined based on the $t$ last values between $n$ and $n-t+1$. Like this, you have $n-t+1$ observations (all except the first few in the time-series because there would not be enough lag observations for them) with $t$ binary variables each (all ordered in the same way starting from the then current value towards the lag values). You can construct all sorts of classifiers based on this data-set, compare their performance and choose the best one to predict. $t$ is a parameter to be tuned.

This will always be a high risk forecast though. Since you basically want to forecast only one number which can only have two values. Per definition, this is going to be all or nothing.

If the costs associated with falsely predicting a 0 are different from the costs of falsely predicting a 1, you can reflect that and bias your algorithm towards predicting more readily the value that would be more expensive to miss.

I may worth to try a neural network for classification, specifically an LSTM is doing quite what you would like to achieve.
It could be used as follows:

• LSTM need input sequences to be the same length. This could be solved by padding the data by adding leading characters. The padding character should be not 0 or 1. Another solution is to use batch size 1 without resetting the status after each batch.
• Once padded data should be encoded using the one hot method. You can use 3 categories: 0, 1 and the padding character.
• The last binary value of each sequence should be used as a target, the rest as input. Of course the padding character will never be the target because we padded on the front.
• Stacking 2 LSTM will do a better job
• Your network could look like: Input->LSTM->LSTM->Dense->Dense->Output
• The more data you have the better it will learn the patterns. Such a network will learn very easily that the padding character is never the output, so don't worry about it.
• I like the idea you're suggesting, however I don't have extensive experience with neural networks in general and LSTM in particular. I will pursue this idea when I have the time though. Commented Aug 30, 2017 at 11:35

Using a Neural Network (NN) would be a solution. NN learn/train on dataset to target dataset, and then make a forecast on that particular database set. NN are very good at processing binary data (some of them generalize input data for better processing).

There is no need to know what binary data represent or how they are translated back (using non-generalized data leads to overwhelming complex models, a beginner overkill).

Some Open Source Statistical Software Packages for Mining Data feature learning NN like Orange/Quasar, JASP, KNIME to name a few, to start on. In general, the more dataset the better outcome. There are more heavy-weight software packages that feature learning NN but there is a huge learning curve not only for starting to use such software packages, but also what in particular all those tools are, what they are for, how and when to use them, and how they perform, especially for a beginner.

The easiest way to start on that in MS Excel is to use XLSTATA, NeuralSolution, PALISADE as MS Excel Add-ins, to name a few, but they are all commercial some offer a full trial use. In Excel such Add-ins offer more easy "control" over steps performed on those tools, like SVM, k-NN, and Log regression, ... to start on.

• What do you mean by "generalize input data for better processing" ? Commented Oct 2, 2022 at 10:03
• Also "NN learn/train on dataset to target dataset, and then make a forecast on that particular database set" sounds weired. Commented Oct 2, 2022 at 10:05