Markov chains #2

Question

A very new European “Rapid Reaction Force for Fire” has been created today and begins operation between three Countries “A”, “B” and “C”. It’s main resource is a super aircraft “Funderbird2” with a massive water cannon that even carries a small mini-submarine for fighting fires at sea. Unfortunately, it can only be in one Country at a time.

What is the matrix representation of this problem?

The probabilities of going to a fire in another Country, given that the force are in a given Country to begin with, are shown on the sketch above (any resemblance to any particular Country or Nation is purely coincidence and not intended). The probabilities were obtained as a weekly average using statistics for fires over many recorded weeks.

When will the plane take up permanent residence in one Country?

Footnote: The final time computed here is the “Laplace transform” of the rate of rescue operations for country C. In this question the matrix approach is used to represent the differential calculus.

This is my homework. I need help from where to start. Would like you to give me directions and to be solved by me, so that I can learn something.

I think I have solved under a) but I am not sure if matrix is correct.

[1.0 0.0 0.0]
[0.1 0.6 0.3]
[0.1 0.4 0.5]

Answer 1

Your transition matrix looks fine to me.

Moving on to your main question, the first thing to understand is that although we are guaranteed to stay in state $A$ if we ever move there, there's no guarantee that we'll actually move there at all. Technically, as the number of iterations of the process approaches infinity the probability of moving to state $A$ converges to $1$, but I don't think that's a satisfactory answer to your question. With that in mind, I suggest rephrasing the problem thusly:

After how many steps is the probability of being in state $A$ greater than some specified probability threshhold $k$ (e.g. $k=0.95$), regardless of starting position?

or alternatively

After how many time steps does the probability of being in state $A$ overwhelm the probability of being in either of the two other states, regardless of starting position?

So how do we determine the probability of being in a given state at time $t$ (i.e. after $t$ iterations)? Since this is homework, I won't give you the answer outright, but I'll try to give you an intuition on how I would go about this. The method I will describe does not require taking the Laplace transform, or really any calculus at all (differential or otherwise), so it may not be what your teacher is driving at. That said:

Your transition matrix gives you the probability of moving to state $j^{(t+1)}$ given that you are in state $i^{(t)}$. Let's consider a particular simulation that starts us in a specific state. Let's say we start in state $B$ at $t=0$. The probability of being in any particular state at $t=1$ is then the second row of your matrix, therefore we can represent our initial position using the vector $i^{(t)} = v=[0,1,0]$, which selects the probability of moving $v \rightarrow j^{(t+1)}$ via: $j^{(t+1)} = v M$ where $M$ is the transition matrix. So now we have a probability distribution over being in any of the three states at $t=1$ given we started in state $B$ via $j^{(1)} = [0.1, 0.6, 0.3]$.

Some questions to help you build your intuition:

1. What is the probability of being in any particular state after two steps given a particular starting position $v$? (hint: I've already shown you how to determine the probability of being in a particular state at $t'=t+1$ given knowledge of your position at time $t$. Now instead of having a fixed position at time $t$ you have a distribution over positions at time $t$, but otherwise the procedure will be the same. The only difference is that earlier we set the probabilities of being in different states as either $0$ or $1$ for time $t$).

2. What is the probability of being in any particular state after $k$ steps given a particular starting position?

3. What is the probability of being in any particular state after $k$ steps if we don't know our starting position?

Hint: this problem can be represented by taking the powers of a matrix.

NB: This solution treats your model as a discrete time chain. I believe the Laplace solution treats the model as a continuous time chain, so my method will always give you a solution in terms of weeks, whereas using an alternative formulation may give you a more granular solution in terms of expected time.