I have 400K observations, each one is a growing binary pattern where order matters. The goal is to predict the likelihood of the next character.

For example, 10101010101? - as humans we can eyeball and say with high confidence that the next character will be a 0.

111111000000111? - in this case, probably a 1.

Any suggestions to the best approach? I was considering expanding the dataset really wide, deriving a lot of lagging variables and so forth, then using regular logistic regression. But I'm curious if there's a better approach.


This problem is similar to predicting the weather from the historical patterns and the states. For example: Today's weather would depend on the previous days, so they should be taken into consideration while calculating the probability of today's weather (the number of states(or the historical data) depends on the analyst).

Such problems can be tackled using the Markov Model. Let me quote the definition from Wikipedia.

In probability theory, a Markov model is a stochastic model used to model randomly changing systems where it is assumed that future states depend only on the present state and not on the sequence of events that preceded it (that is, it assumes the Markov property). Generally, this assumption enables reasoning and computation with the model that would otherwise be intractable.

I want to give you a detailed answer about how the Markov models would work and look, for solving a problem similar to yours. But, seems like this article has done a fairly neat job at it.

An example piece of literature which you would be interested in.

References: Markov Models in the Analysis of Frequent Patterns in Financial Data by Julija PRAGARAUSKAITE, Gintautas DZEMYDA


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.