# Can a 1st Order Model capture information that a 2nd Order model can't?

Let's say we have some sequence $y_1 \ldots y_n$. By definition, a 2nd order Markov model can capture more information from the sequence than a first or zeroth order Markov model. What I'm interested in, however, is if the first order model can capture some information that the second order model can't.

The reason I ask is the following. I have a procedure where, given a sequence with certain annotations, I learn a second order model from that sequence and use it to make predictions. These predictions are then used as input to the same procedure, and the process continues until the predictions don't change.

It turns out, however, that if I first do the above process using a first order model for several iterations, then use that output as an input for the same process with a second order model, I get even better results.

If I also add a zeroth order model before that, I get even better results.

My intuition tells me that the lower order models somehow filter out noise (i.e. false correlations that the second order model might capture), but I can't seem to find a good reason as to why this would happen.

Any suggestions? To those interested, the specific application is gene-start prediction.

• What do you mean by "use that output"? I can think of several alternatives... Commented Sep 29, 2014 at 23:12
• @Just, maybe you can describe the estimation procedure of the models more explicitly, eg, is it optimization-based? Is this a hidden Markov model? Commented Sep 30, 2014 at 8:33

This is a typical overfitting situation. Your 2nd order class model contains the 1st order model, so in theory it should do better, but due to the finiteness of your training data, you are getting 'noisy' parameters that should have been zeros. This happens all the time, and the usual solution is to regularize the model (unless you are already doing so).

• Note that I still run the second order model in the second case. The only difference is that I start off using a first order model and move on to a second order. If overfitting occurs, shouldn't both cases express it in the same way? Both ultimately use second order models.
– Just
Commented Sep 29, 2014 at 19:58