Questions tagged [transition-matrix]

A transition matrix is a square matrix used to describe the transitions of a Markov chain.

Filter by
Sorted by
Tagged with
1
vote
0answers
16 views

How to maximize the steady state transition probability for a state in a Markov chain by altering that state's outgoing transition probabilities?

Let's say we have a transition matrix of which can be solved to come up with steady state transition probabilities of Alpha: 34.9% Beta: 24.0% Gamma: 16.9% Delta: 24.2% Now, imagine Alpha, Beta, ...
0
votes
0answers
16 views

Simulate discrete state space CTMC from generator matrix

Consider a generator matrix $Q\in\mathbb{R}^{h\times h}$ for a discrete state space $\{1,...,h\}$. I want to determine the probability of a single transition of a stochastic process $X(t)$ with $X(0)=...
0
votes
0answers
15 views

Creating a transition matrix based on a Markov chain in R

I have four distributions that represent incomes in R. I categorise them by what income group they fall under such as under half the mean, between half the mean and 3/4th of the mean and so on until ...
0
votes
0answers
10 views

Transition probablities vs transition rate

What would you describe and differentiate between Transition rate and probabilities intuitively in accordance with Transition probablity matrix as well as markov chains and HSD models.
0
votes
2answers
19 views

How to calculate the probability Matrix (Alpha) for Regular Markov chains

Pardon me for being a novice here. In the image attached, eq 3.1 represents the transition matrix (it's pretty clear). I am not able to comprehend the eq 3.2, alpha*P = alpha, as well as the further ...
0
votes
0answers
40 views

Accuracy in Estimating Customer Lifetime Value using Markov Chain Model

I've an online customer data which has the purchases made in every month and recency of the purchases information for 12 months. So data looks like below: ...
0
votes
1answer
64 views

$\pi_i P^n_{i, j} =$ long-run proportion of time the chain is in $i$ and will be in $j$ after $n$ transitions?

I am currently studying the textbook Introduction to Probability Models by Sheldon M. Ross. Chapter 4.4 Long-Run Proportions and Limiting Probabilities says the following: Because $\pi_i$ is the long-...
0
votes
1answer
27 views

generate a realization from a transition matrix

Consider a markov chain of 4 states $\{S_1, S_2, S_3, S_4\}$ described by the transition matrix $$ A = \begin{bmatrix} .25 & .20 & .25 & .30 \\ .20 & .30 & .25 & .30 \\ ....
1
vote
1answer
22 views

Calculating limit law for matrix

I my notes on Markov chains, I am presented with the following matrix: $$\mathcal{P} = \begin{bmatrix} 0.97 & 0.03 & 0 & 0 \\ 0.008 & 0.982 & 0.01 & 0 \\ 0.02 & 0 & 0....
1
vote
1answer
51 views

Example where unique stationary law, which is an occupation law, but no limit law exists

I am currently learning about the balance equations, mass equation, limit law, occupation law and stationary law in Markov models. The following example is presented: Example 2: $$\mathcal{P} = \...
1
vote
1answer
24 views

Computation of balance equation example in Markov model

I am studying some examples of balance equations for Markov models. I am presented with the following example: $$\mathcal{P} = \begin{bmatrix} 0.2 & 0.3 & 0.5 \\ 0.1 & 0 & 0.9 \\ 0.55 ...
1
vote
1answer
27 views

Markov models and occupation time

I'm presented with the following explanation and proof: Let $(X_n)$ be a Markov chain, and fix a state $j \in S$. Define indicator variables: For $n = 0, 1, \dots$, let $$I_n(j) = \begin{cases} 1 &...
1
vote
1answer
24 views

Calculating 7-step transition matrix for example

In my notes on Markov processes, I am presented with two related examples: Example 1: Classify daily weather for some region as Sunny (state $1$), Cloudy (state $2$), or rainy (state $3$). ...
0
votes
1answer
37 views

Proof that the $n$-step transition matrix is the $n$th power of $\mathcal{P}$

I am presented with the following theorem in the context of Markov chains and stochastic systems: The $n$-step transition matrix is the $n$th power of $\mathcal{P}$: $$\mathcal{P}^{(n)} = P^n.$$ And ...
1
vote
0answers
16 views

Transition probability matrix

A gambler tosses a coin and a tetrahedron at each stage. If $H$, he receives the amount appearing at the face of the tetrahedron. If $T$, he pays the amount. The tetrahedron is fair, but probability ...
0
votes
0answers
14 views

Find transition probability matrix

A box contains 3 balls. Each is either white or red. The game is to draw a ball from each period. If red is drawn, a white is replaced. But if white is drawn, all the balls in the box is replaced by ...
0
votes
0answers
20 views

Reducible Markov chain with one of the pi entires in the pi vector as zero

I have a reducible, finite markov chain with 2 absorbing states, when finding the stationary distribution I got one of the pi's of the pi vector to be zero. Does this mean that the Markov chain does ...
0
votes
1answer
36 views

Number of stationary distributions of a Markov chain

How do i determine the number of stationary distributions that a Markov chain has if it is not irreducible or regular. The transition matrix is ...
2
votes
1answer
149 views

Estimating model for transition probabilities of a Markov Chain

Suppose that I have a Markov chain with $S$ states evolving over time. I have $S^2\times T$ values of the transition matrix, where $T$ is the number of time periods. I also have $K$ matrices $X$ of $T\...
4
votes
1answer
111 views

How to show that the transition probability is equal to $\overline p_{ij} = \frac{P_{ij}}{\sum_{k\neq i}p_{ik}}$

(No new answers needed) I would like to award @whuber for his good answer with my bounty! Suppose that $(X_n)_{n≥0}$ is Markov$(λ, P)$ but that we only observe the process when it moves to a new ...
1
vote
1answer
49 views

Rows and columns of the one-step transition probability matrix

I am currently studying the textbook Introduction to Modeling and Analysis of Stochastic Systems, Second Edition, by V. G. Kulkarni. In a section on discrete-time Markov chains, the author introduces ...
0
votes
0answers
34 views

Singular state transition probability matrix in David Silver's UCL Lesson 2

I'm studying David Silver's second lesson on reinforcement learning: https://www.youtube.com/watch?v=lfHX2hHRMVQ&list=PLqYmG7hTraZDM-OYHWgPebj2MfCFzFObQ&index=2 and the state transition ...
1
vote
0answers
14 views

Specifying a multi-state model with unobservable terminal state

Suppose there is a multi-state process with three states, listed below and labelled as terminal/non-terminal and observable/unobservable: Initializing: non-terminal observable Active: non-terminal ...
0
votes
0answers
129 views

Make sense of plotting a transition matrix

I'm studying statistics and I'm trying to understand markov chain topic. I'm using the package "markovchain" in R to obtain the stationary distribution. From this transition matrix $M$: <...
1
vote
1answer
245 views

Measure the distance between two probability transition matrices

I have a probability transition matrix $P$ that contains some values very close to zero. I want to sparsify this matrix by taking the k largest values for each row and setting the others to zero. For ...
0
votes
1answer
52 views

Long run proportion of transitions in a Markov chain

Let $S$ be a set of states for a Markov chain and let $S^C$ be the remaining states. Explain the identity $$\sum_{i\in S}\sum_{j\in S^C}\pi_iP_{ij}=\sum_{i\in S^C}\sum_{j\in S}\pi_iP_{ij}$$ I know ...
1
vote
1answer
231 views

Irreducible (communicating) classes [closed]

The Markov chain $(Xn; n\geq)$ has state-space $S = (0, 1, 2, . . .)$, with $p_{i,0} = \frac{1}{4}$ and $p_{i,i+1} = \frac{3}{4}$ $\forall i \geq 0$, so that the transition matrix is P =$\...
0
votes
1answer
139 views

Biased coins and Markov processes

Good day, I am attempting an optional exercise and I am finding it hard to interpret the problem in terms of matrices and vectors. Coin 1 has probability 0.4 of coming up heads, and coin 2 has ...
0
votes
0answers
101 views

Formulating a Transition matrix for Markov Process

I am dealing with a medical process which is as follows. There are 10000 Veterans who are enrolled in this study. All 10000 have medical condition called onychocryptosis which is a fancy term for ...
2
votes
1answer
303 views

What is the difference betwen a time non-homogenous Markov Chain and a non-linear Markov Chain? Example

A time non-homogenous Markov Chain is one in which the transition probabilities are not constant over time. A non-linear Markov Chain is a model that is not linear in parameters and satisfies the ...
1
vote
0answers
434 views

Irreducible Markov chain and transition matrix

We know that a matrix is irreducible if it is not similar via a permutation to a block upper triangular matrix. Is the transition matrix of a irreducible Markov chain irreducible?
1
vote
0answers
174 views

Bayesian inference of parameter governing Markov transition matrix

A 3-state Markov chain $X = \{x_i : i \in \{1, \cdots, N\}\}$ is observed, and its transition matrix $P$ is assumed to be of the form $$ \begin{pmatrix} (1-a)^2 & 2a(1-a) & a^2 \\ b(1-a) &...
1
vote
0answers
116 views

Bayesian inference of non-homogeneous Markov transition matrix

The data consists of several discrete-time Markov chains, indexed by a global time. I assume all the chains are governed by the same transition matrix, but that this can change in time. I want to ...
2
votes
0answers
318 views

A Hidden Markov model with covariates in the transition probabilities

I would like to construct a Hidden Markov model with data about online customer journeys. A well-known concept related to the customer journey literature is the sales funnel. Consumers walk through ...
0
votes
0answers
78 views

How two transition probability matrices can be used to find one variable in analysis

I'm working on a research paper for cricket analysis but I'm stuck on the point of the estimation of $\tau_{owj}$, which is defined in the appendix. I have calculated up through $\alpha_{owj}$ and $\...
0
votes
0answers
588 views

Second order markov tranision probability matrix

I tried to find the second order Markov chain of the following sequence Dat= A A B A B A A A B B A A B I tried it on "Markov chain" package in R. ...
0
votes
2answers
502 views

Markov-Chain transition probabilities for 3 variables

I am a bit confused as I need to calculate the Markov-Chain transition probabilites for 3 variables. Example data, let's assume a sequence of letters at specific and progressively-constant time steps:...
0
votes
1answer
595 views

Seeking examples of latent transition analysis with ordinal data and multiple subjects

Background A colleague of mine has asked me for help. She has a large amount of patient data involving clinical psychological measures (e.g., questions related to specific symptoms) gathered both ...
0
votes
1answer
815 views

MLE for Markov Chains - intuitive explanation

could anyone please give me intuitive explanation what does below mean ? Let say I have sequence: a, b, a, b, b, b, a, b, b, a By applying Markov Chains with MLE method in R package on below I get ...
0
votes
1answer
306 views

Irreducible Markov Chain Question

If you have an irreducible Markov chain with transition matrix $P$, and $p(j,j) > 0$ for all $j$, why are all states aperiodic?
1
vote
1answer
792 views

“Symmetric” property of stationary distribution

The above symmetric property isn't referring to the double stochastic property \begin{bmatrix} 0.2&0.8&0&0&0\\ 0.2&0.2&0.6&0&0\\ 0&0.4&0.2&0.4&0\\ 0&...
0
votes
1answer
1k views

Expected value in Markov chains

Let $\left\{X_{n}\right\}_{n\geq0}$ be a homogeneous markov chain with state space E and transition matrix P. Let $\tau$ be the first time n for which $X_{n}$ $\neq$ $X_{0}$, where $\tau=+\infty$ if $...
1
vote
0answers
64 views

How to find certain probabilities in transition matrices

A unit consists of three circuit elements in parallel; it will be in working order as long as at least one of the elements is working. The unit is examined every hour and if it is not in working order ...
0
votes
0answers
159 views

How to bet on a binary event based on the markov transition matrix, state probabilities and the odds

There is a coupon full of football matches for a given day from a bookkeeper. I have scrapped another website and i have aquired continuous history of a particular match between ...
3
votes
1answer
3k views

How to translate an adjacency matrix to a transition matrix for use in Markov cluster algorithm? [closed]

I have a matrix of size (47*47 double) that have only 0's and 1's. I want to apply the Markov clustering algorithm on this matrix, but this Method needs a transition matrix as the columns must be ...
4
votes
0answers
315 views

Find the invariant measure $\pi=(\pi_{1},\pi_{2},\pi_{3})$ for a Markov Chain with transition matrix given

Let $(X_{n})_{n\in\mathbb{N}_{0}}$ be a Markov Chain with state space $M=\left\{x_{1},x_{2},x_{3}\right\}$ and transtition matrix $$ \Pi=\left(\begin{array}{ccc}p_{1} & p_{2} & 1-p_{1}-p_{2}\\ ...
0
votes
0answers
33 views

Calculate the transition matrix of $X_{n+1}:= \sum_{i=0}^{X_{n}}\theta_{n}^{i}\:\: \mbox{mod }5.$ where $\theta_{n}^{i}\sim Bin(3,1/3)$ i.i.d

Given an i.i.d. sequence $(\theta_{n}^{i})_{n,i\in\mathbb{N}}$ with binomial distribution $\mathcal{B}_{3,\frac{1}{3}}$ we define the Markov Chain $$X_{n+1}:= \sum_{i=0}^{X_{n}}\theta_{n}^{i}\:\:\: \...
4
votes
1answer
977 views

Constructing a transition probability from Q-learning

In Reinforcement learning, learning without the need for the transition probability matrix is 'model free learning'. Instead of having the transition probabilities, we learn the q-values (state/action ...
1
vote
0answers
34 views

Markov process with conditions

I am trying to model a basketball game using play by play data. Each team has a transition matrix representing what they will do next on offense and defense in any given time remaining in the game. ...
2
votes
1answer
676 views

Finding steady-state probability of a Markov chain

Let $X_{n}$ be a Markov chain on state space $S = \{ 1,2 \dots, 23 \}$ with transition probability given by $p_{i,i+1}= p_{i,i-1} = \frac {1}{2} \ \ \forall \ 2\le i \le 22 , $ $ p_{1,2}= p_{1,23} =...