Let's say we have some sequence $y_1 \ldots y_n$. By definition, a 2nd order Markov model can capture more information from the sequence than a first or zeroth order Markov model. What I'm interested in, however, is if the first order model can capture some information that the second order model can't.
The reason I ask is the following. I have a procedure where, given a sequence with certain annotations, I learn a second order model from that sequence and use it to make predictions. These predictions are then used as input to the same procedure, and the process continues until the predictions don't change.
It turns out, however, that if I first do the above process using a first order model for several iterations, then use that output as an input for the same process with a second order model, I get even better results.
If I also add a zeroth order model before that, I get even better results.
My intuition tells me that the lower order models somehow filter out noise (i.e. false correlations that the second order model might capture), but I can't seem to find a good reason as to why this would happen.
Any suggestions? To those interested, the specific application is gene-start prediction.