# Different "cases" in Markov chain example

I am presented with the following Markov chain example:

A trader sells large and expensive machines.

$$X_n$$ is the number of machines in stock at the start of week $$n$$.

$$D_n$$ is the number of machines demanded by customers during week $$n$$.

Assume that $$D_n \sim \text{Poi}(3)$$, the $$D_n$$ are independent, and that $$D_n$$ and $$X_n$$ are independent for each $$n$$.

There are two stipulations:

1. Inventory control: If $$0$$ or $$1$$ machines are left in stock by the end of a week, then machines are ordered and delivered to raise the stock to $$5$$ by the start of the next week. If $$2$$ or more machines are in stock at the end of the week, then no orders are placed.

2. Lost business: If $$D_n > X_n$$, then the unsatisfied demands are lost.

We seek to show that $$X_n$$ is a Markov chain.

$$(X_n - D_n)^+$$ is the number of machines in stock at the end of week $$n$$.

$$X_{n + 1} =\begin{cases} X_n - D_n & \text{if} X_n - D_n \ge 2 \\ 5 & \text{if} X_n - D_n \le 1 \end{cases}$$

$$S = \{ 2, 3, 4, 5 \}$$

Denote the history of the process up to time $$n$$ by $$H_n = \{ X_0, X_1, \dots, X_n \}$$.

Given that $$X_n = i$$, the independence assumptions ensure that $$X_{n + 1}$$ is conditionally independent of $$H_{n - 1}$$.

Case 1: $$j = 2, 3,$$ or $$4$$, and $$i = j, \dots, 5$$:

\begin{align} P(X_{n + 1} = j \vert X_n = i, H_{n - 1}) &= P(X_n - D_n = j \vert X_n = i, H_{n - 1}) \\ &= P(i - D_n = j \vert X_n = i, H_{n - 1}) \\ &= P(i - D_n = j) \\ &= P(D_n = i - j) \\ &= e^{-3} \dfrac{3^{i - j}}{(i - j)!} \end{align}

Case 2: $$i = 2, 3,$$ or $$4$$, and $$j = 5$$:

\begin{align} P(X_{n + 1} = 5 \vert X_n = i, H_{n - 1}) &= P(X_n - D_n \le 1 \vert X_n = i, H_{n - 1}) \\ &= P(i - D_n \le 1 \vert X_n = i, H_{n - 1}) \\ &= P(i - D_n \le 1) \\ &= P(D_n \ge i - 1) \\ &= 1 - P(D_n \le i - 2) \\ &=: p_{i5} \end{align}

Case 3: $$i = 5, j = 5$$:

$$p_{55} = P(D_n = 0) + P(D_n \ge 4) = P(D_n = 0) + 1 - P(D_n \le 3).$$

Can someone explain the different "cases" here? Specifically, I'm unsure about of the reasoning the led to the three cases.

I would greatly appreciate it if people would please take the time to clarify this.

We have $$X_{n + 1} =\begin{cases} X_n - D_n & \text{if} X_n - D_n \ge 2 \\ 5 & \text{if} X_n - D_n \le 1 \end{cases} \tag{*}$$

The two rules of $$(*)$$ means no restocking and restocking respectively.

We want $$X_n$$ to take value $$i$$ and $$X_{n+1}$$ to take value $$j$$ where $$i,j \in \{2,3,4,5\}$$.

From $$(*)$$, it is natural to handle the case where $$j$$ takes value $$5$$ separately because the remaining case where $$j<5$$ can only come from the first rule, the no restocking rule.

• Hence in case $$1$$, we first address the case where $$j<5$$, which can only comes from the first rule in $$(*)$$. Hence we need $$i \ge j$$ since $$D_n$$ is a nonnegative random variable.

Case 1: $$j = 2, 3,$$ or $$4$$, and $$i = j, \dots, 5$$:

\begin{align} P(X_{n + 1} = j \vert X_n = i, H_{n - 1}) &= P(X_n - D_n = j \vert X_n = i, H_{n - 1}) \\ &= P(i - D_n = j \vert X_n = i, H_{n - 1}) \\ &= P(i - D_n = j) \\ &= P(D_n = i - j) \\ &= e^{-3} \dfrac{3^{i - j}}{(i - j)!} \end{align}

Now, for the remaining case, we just have to address when $$j$$ takes value $$5$$.

• We note that if $$i < 5$$, the only way for us to end up with $$5$$ is via restocking, that is we have to use the second rule. This is our case $$2$$.

Case 2: $$i = 2, 3,$$ or $$4$$, and $$j = 5$$:

\begin{align} P(X_{n + 1} = 5 \vert X_n = i, H_{n - 1}) &= P(X_n - D_n \le 1 \vert X_n = i, H_{n - 1}) \\ &= P(i - D_n \le 1 \vert X_n = i, H_{n - 1}) \\ &= P(i - D_n \le 1) \\ &= P(D_n \ge i - 1) \\ &= 1 - P(D_n \le i - 2) \\ &=: p_{i5} \end{align}

• Now the only remaining case that we haven't covered is when $$(i,j)=(5,5)$$, where it can be due to there is no demand that we have to use the first rule or there is too much demand that we have to restock and we have to use the second rule.

Case 3: $$i = 5, j = 5$$:

$$p_{55} = P(D_n = 0) + P(D_n \ge 4) = P(D_n = 0) + 1 - P(D_n \le 3).$$