12
$\begingroup$

First, let me acknowledge up front that I'm not as well versed in statistics and mathematics as I'd like to be. Some might say have just enough knowledge to be dangerous. :D I apologize if I'm not using terminology correctly.

I am trying to model the probabilities of a system transitioning from one state to another. A simple Markov model is a good start. (Set of states, set of initial state probabilities, set of transition probabilities between states.)

However, the system I'm modeling is more complex than that. The transition probabilities leading to a state at time T are most certainly dependent on variables other than the state at T-1. For example, S1 -> S2 might have a transition probability of 40% when the sun is shining, but S1 -> S2 probability goes to 80% when it is raining.

Additional info from commenters' questions:

  1. The states are observable.
  2. There will only be 5-10 states.
  3. There are currently about 30 covariates that we want to investigate, though the final model will certainly have fewer than this.
  4. Some covariates are continuous, others are discrete.

Three questions:

  1. How can I incorporate conditional transition probabilities into my Markov model?
  2. Or, is there another perspective entirely from which I should approach this issue?
  3. Also, what keywords/concepts should I search for online to learn more about this?

I've already been around the web searching for things like "markov models with conditional transition probabilities," but so far nothing has slapped me in the face and said, "This is your answer, dummy!"

Thank you for your help and patience.

Improve this question
$\endgroup$
2
  • $\begingroup$ Welcome to the site. How large is the state space? Do you observe the state that your process is in at each step? How many covariates (additional predictors) do you have? Are they continuous, discrete or perhaps a mixture of both? $\endgroup$
    – cardinal
    Commented Mar 9, 2012 at 18:07
  • $\begingroup$ Thanks, cardinal. Yes, the states are observable. There will probably be 5 to 10 states. (It's still uncertain, but I don't expect a very large state space.) Right now, we have a list of about 30 additional covariates that we intend to investigate, although most of them will probably end up having little effect. Some are continuous, and some are discrete. $\endgroup$
    – Aaron Johnson
    Commented Mar 9, 2012 at 19:00

3 Answers 3

Reset to default
6
$\begingroup$

You can always have a 2nd order or higher order markov chain. In that case your model all ready includes all probabilistic transition information in it. You can check Dynamic Bayesian Networks which is a graphical model generalization of Markov Chains that are utilized frequently in machine learning.

Improve this answer
$\endgroup$
5
  • $\begingroup$ YBE, Thanks for the quick reply! Does this (modeling the system as a 2nd order or higher chain) allow me to model continuous covariates, or just discrete covariates? And can you point me to a link that gives a good example of what you're talking about? Thanks! $\endgroup$
    – Aaron Johnson
    Commented Mar 12, 2012 at 6:04
  • $\begingroup$ There is a paper you can check. It first starts describing 1st order chains, then describes the situation for higher order chains. (Higher-order multivariate Markov chains and their applications by Ching,Ng,Fung) If you are interested in machine learning kind of stuff, I suggest you to check Kevin Murphy's website. He also has a MATLAB toolbox that you can play with. $\endgroup$
    – jkt
    Commented Mar 12, 2012 at 6:27
  • $\begingroup$ +1 to your answer for the reference to the Ching, Ng, and Fung paper. That's a good one to have. However, after reading through it, it appears that it only covers discrete variables (which is kind of what I expected.) While I can discretize my continuous variables, I'm still curious -- Are there any models that can handle the raw continuous variables? $\endgroup$
    – Aaron Johnson
    Commented Mar 19, 2012 at 16:02
  • $\begingroup$ I am not an expert, but I guess the results should hold for continuous case in general. Kalman filter for instance runs on an HMM (1st order markov chain) with continuous states. $\endgroup$
    – jkt
    Commented Mar 19, 2012 at 20:12
  • $\begingroup$ I didn't immediately pick your answer because I was waiting for more candidates. They never came, and I forgot about it. Two years later, I now award you by accepting your answer. Thanks for the info! By the way, have you come across anything else on this topic in the last two years? It's still something that I'm interested in. $\endgroup$
    – Aaron Johnson
    Commented Jul 30, 2014 at 14:42
2
$\begingroup$

I believe what you are looking for is Maxent Markov Models.

Or you may used a generalisation (if I understand it right) of Maxent Markov models, which are called Conditional Random Fields.

Improve this answer
$\endgroup$
0
$\begingroup$

I was asking myself the same question and if you really only need to model the outcome based on the state at $T_1$ and covariates, you may find the msm package in R helpful.

This package seems to be a pretty good fit for modeling the effects of covariates on transitions between categorical outcomes over time. It wouldn't help if you really do need a higher order chain, but it doesn't seem like that's the case based on your original question.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.