I am trying to simulate a process as an absorbing Markov chain model, but I haven't been able to find the scenario that I am interested in looking at in the typical discussions of Markov chains online. Hoping someone out there will have some pointers.
In the chain I am building, there are certain transitions between states that have a certain utility or value. Values are always non-negative. The majority of transitions have zero value, but the few that have values are what's important in this analysis. I am interested in the expected sum of values from all transitions until the chain reaches an absorbing state (there could be more than one absorbing state).
In other words, as the chain evolves, it will pass through certain transitions that have value > 0, and I want to know the expected value the chain accumulates before it ends in an absorbing state.
Is there standard terminology for what I'm trying to describe? What would be the best way to go about finding this value?
I'd also be interested in the probability distribution of which absorbing state the chain terminates in, although that's less important that the expected value.
I will need to test numerous Markov chains with the same set of states, but different transition probabilities to see how the expected value changes.
One strategy that occurs to me is that I could change the transitions that have positive values to a transition into/out of a dummy state, so if I could count the number of expected visits to each dummy state, I could weight them by the values of their transitions appropriately and get the expected value that way. So an alternate form of the question might be, how can you count the expected number of visits to a particular state in an absorbing Markov chain?
I would prefer to implement this in R, especially if there's a package that already does some portion of this. OpenBUGS might be an option too if it makes sense (through R via package rbugs, although I haven't used any BUGS software before).
Any guidance on how to approach this problem or recommendations on tools to solve it would be appreciated.