Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [transition-matrix]

A transition matrix is a square matrix used to describe the transitions of a Markov chain.

0
votes
1answer
23 views

Long run proportion of transitions in a Markov chain

Let $S$ be a set of states for a Markov chain and let $S^C$ be the remaining states. Explain the identity $$\sum_{i\in S}\sum_{j\in S^C}\pi_iP_{ij}=\sum_{i\in S^C}\sum_{j\in S}\pi_iP_{ij}$$ I know ...
0
votes
0answers
44 views

Transition probability in Markov chain using a decision tree model

I wish to find a way to calculate the transition probabilities in my Markov chain model. Let's say a customer has three products [A B C] and in this example I wish to know the transition probability ...
1
vote
1answer
66 views

Irreducible (communicating) classes [closed]

The Markov chain $(Xn; n\geq)$ has state-space $S = (0, 1, 2, . . .)$, with $p_{i,0} = \frac{1}{4}$ and $p_{i,i+1} = \frac{3}{4}$ $\forall i \geq 0$, so that the transition matrix is P =$\...
0
votes
1answer
60 views

Biased coins and Markov processes

Good day, I am attempting an optional exercise and I am finding it hard to interpret the problem in terms of matrices and vectors. Coin 1 has probability 0.4 of coming up heads, and coin 2 has ...
0
votes
0answers
72 views

Formulating a Transition matrix for Markov Process

I am dealing with a medical process which is as follows. There are 10000 Veterans who are enrolled in this study. All 10000 have medical condition called onychocryptosis which is a fancy term for ...
2
votes
1answer
119 views

What is the difference betwen a time non-homogenous Markov Chain and a non-linear Markov Chain? Example

A time non-homogenous Markov Chain is one in which the transition probabilities are not constant over time. A non-linear Markov Chain is a model that is not linear in parameters and satisfies the ...
0
votes
0answers
40 views

Estimation of infinitesimal generator/transition rate matrix from proportion data

Suppose I have a collection of data $\{\boldsymbol x_t \in \mathbb S^d\}_{t = 1,\dots,T}$ where $\mathbb S^d$ is the $d$-dimensional unit simplex, i.e. the elements of $\boldsymbol x_t$ sum to $1$. ...
0
votes
0answers
59 views

How to identify irreducible states by looking at a markov transition matrix?

I'm trying to find a simple way to look at a markov transition matrix and determine the subset of the states which form a closed, irreducible set of states. I came up with the following: If a set of ...
1
vote
0answers
298 views

Irreducible Markov chain and transition matrix

We know that a matrix is irreducible if it is not similar via a permutation to a block upper triangular matrix. Is the transition matrix of a irreducible Markov chain irreducible?
0
votes
0answers
93 views

Estimate Transition Probabilities for a Markov Model from clinical trial data

I'm working on Markov Model for a healthcare problematic. Actually, I have data for a clinical trial (https://clinicaltrials.gov/ct2/show/NCT01006980) but it shows the number of patients moving from ...
1
vote
0answers
99 views

Bayesian inference of parameter governing Markov transition matrix

A 3-state Markov chain $X = \{x_i : i \in \{1, \cdots, N\}\}$ is observed, and its transition matrix $P$ is assumed to be of the form $$ \begin{pmatrix} (1-a)^2 & 2a(1-a) & a^2 \\ b(1-a) &...
0
votes
0answers
25 views

Using Lift to filter Transition Matrix for Word2Vec

Can we equate likelihood $P( A | B )$, to non-self cyclical, i.e.: $P ( A | A ) = 0$ transition matrix in page rank? If yes, does it make sense to ignore pairs with $\text{lift} \le 1$, where $$\text{...
1
vote
0answers
84 views

Bayesian inference of non-homogeneous Markov transition matrix

The data consists of several discrete-time Markov chains, indexed by a global time. I assume all the chains are governed by the same transition matrix, but that this can change in time. I want to ...
0
votes
1answer
154 views

A Hidden Markov model with covariates in the transition probabilities

I would like to construct a Hidden Markov model with data about online customer journeys. A well-known concept related to the customer journey literature is the sales funnel. Consumers walk through ...
0
votes
0answers
62 views

How two transition probability matrices can be used to find one variable in analysis

I'm working on a research paper for cricket analysis but I'm stuck on the point of the estimation of $\tau_{owj}$, which is defined in the appendix. I have calculated up through $\alpha_{owj}$ and $\...
0
votes
0answers
389 views

Second order markov tranision probability matrix

I tried to find the second order Markov chain of the following sequence Dat= A A B A B A A A B B A A B I tried it on "Markov chain" package in R. ...
0
votes
2answers
322 views

Markov-Chain transition probabilities for 3 variables

I am a bit confused as I need to calculate the Markov-Chain transition probabilites for 3 variables. Example data, let's assume a sequence of letters at specific and progressively-constant time steps:...
0
votes
1answer
375 views

Seeking examples of latent transition analysis with ordinal data and multiple subjects

Background A colleague of mine has asked me for help. She has a large amount of patient data involving clinical psychological measures (e.g., questions related to specific symptoms) gathered both ...
0
votes
1answer
569 views

MLE for Markov Chains - intuitive explanation

could anyone please give me intuitive explanation what does below mean ? Let say I have sequence: a, b, a, b, b, b, a, b, b, a By applying Markov Chains with MLE method in R package on below I get ...
0
votes
1answer
283 views

Irreducible Markov Chain Question

If you have an irreducible Markov chain with transition matrix $P$, and $p(j,j) > 0$ for all $j$, why are all states aperiodic?
1
vote
1answer
526 views

“Symmetric” property of stationary distribution

The above symmetric property isn't referring to the double stochastic property \begin{bmatrix} 0.2&0.8&0&0&0\\ 0.2&0.2&0.6&0&0\\ 0&0.4&0.2&0.4&0\\ 0&...
0
votes
1answer
844 views

Expected value in Markov chains

Let $\left\{X_{n}\right\}_{n\geq0}$ be a homogeneous markov chain with state space E and transition matrix P. Let $\tau$ be the first time n for which $X_{n}$ $\neq$ $X_{0}$, where $\tau=+\infty$ if $...
1
vote
0answers
55 views

How to find certain probabilities in transition matrices

A unit consists of three circuit elements in parallel; it will be in working order as long as at least one of the elements is working. The unit is examined every hour and if it is not in working order ...
0
votes
0answers
74 views

How to bet on a binary event based on the markov transition matrix, state probabilities and the odds

There is a coupon full of football matches for a given day from a bookkeeper. I have scrapped another website and i have aquired continuous history of a particular match between ...
3
votes
1answer
2k views

How to translate an adjacency matrix to a transition matrix for use in Markov cluster algorithm? [closed]

I have a matrix of size (47*47 double) that have only 0's and 1's. I want to apply the Markov clustering algorithm on this matrix, but this Method needs a transition matrix as the columns must be ...
4
votes
0answers
248 views

Find the invariant measure $\pi=(\pi_{1},\pi_{2},\pi_{3})$ for a Markov Chain with transition matrix given

Let $(X_{n})_{n\in\mathbb{N}_{0}}$ be a Markov Chain with state space $M=\left\{x_{1},x_{2},x_{3}\right\}$ and transtition matrix $$ \Pi=\left(\begin{array}{ccc}p_{1} & p_{2} & 1-p_{1}-p_{2}\\ ...
0
votes
0answers
27 views

Calculate the transition matrix of $X_{n+1}:= \sum_{i=0}^{X_{n}}\theta_{n}^{i}\:\: \mbox{mod }5.$ where $\theta_{n}^{i}\sim Bin(3,1/3)$ i.i.d

Given an i.i.d. sequence $(\theta_{n}^{i})_{n,i\in\mathbb{N}}$ with binomial distribution $\mathcal{B}_{3,\frac{1}{3}}$ we define the Markov Chain $$X_{n+1}:= \sum_{i=0}^{X_{n}}\theta_{n}^{i}\:\:\: \...
2
votes
1answer
724 views

Constructing a transition probability from Q-learning

In Reinforcement learning, learning without the need for the transition probability matrix is 'model free learning'. Instead of having the transition probabilities, we learn the q-values (state/action ...
1
vote
0answers
34 views

Markov process with conditions

I am trying to model a basketball game using play by play data. Each team has a transition matrix representing what they will do next on offense and defense in any given time remaining in the game. ...
3
votes
1answer
568 views

Finding steady-state probability of a Markov chain

Let $X_{n}$ be a Markov chain on state space $S = \{ 1,2 \dots, 23 \}$ with transition probability given by $p_{i,i+1}= p_{i,i-1} = \frac {1}{2} \ \ \forall \ 2\le i \le 22 , $ $ p_{1,2}= p_{1,23} =...
3
votes
4answers
1k views

Essential transient state in a Markov chain

Can a finite state Markov chain have essential transient state? I have found out an example for an infinite state one and I have the intuition (I may be wrong) that for a finite state space .. This ...
3
votes
0answers
273 views

The expected long run proportion of time the chain spends at $a$ , given that it starts at $c$

Consider the transition matrix: $\begin{bmatrix} \frac{1}{5} & \frac{4}{5} & 0 & 0 & 0 \\ \frac{1}{2} & \frac{1}{2} & 0 & 0 & 0 \\ \frac{1}{5} & \frac{1}{5} & \...
0
votes
0answers
121 views

Stationary Probabilities

A transition probability matrix $P$ is said to be doubly stochastic, if each column sum is 1;that is if $\displaystyle\sum_i P_{i,j} = 1$ for all j If such a chain is irreducible and have states $1,...
0
votes
0answers
172 views

When Markov Chain goes to stable?

Suppose I have a transition matrix $A$, which is positive definite. I know that this chain will goes to stable after some finite time. The question is when will this chain goes to stable? Or, after ...
7
votes
0answers
300 views

Reinforcement *Model* Learning

Classical reinforcement learning (Q- or Sarsa-Learning) can be extended with models of the environment. These models are usually transition tables that contain the probability of arriving at a ...
0
votes
1answer
579 views

Regular Transition Matrix

A transition matrix is regular if some power of the matrix contains all positive entries. [1] Its powers have all positive entries... Why isn't this matrix a Regular Transition Matrix? Reference: ...
1
vote
0answers
295 views

how to calculate transition probability matrix for n*4 data

data: https://www.dropbox.com/s/4ni96jmzn14sjl4/florep_normn4_p1.mat?dl=0 I attached n*4 data. so here each row has 4 values and each row considered as one state. I want to calculate transition ...
1
vote
1answer
298 views

Transition matrix calculation from text book “Introduction to information retrieval”

According to the example on page in the text book, the graph of documents is as follows: $1\rightarrow2, 3\rightarrow2, 2\rightarrow1, 2\rightarrow3$ with $\lambda = 0.5$ which would form an ...
2
votes
1answer
704 views

Markov chain transition matrix and state vector order?

The typical way of performing state simulations using matrices is to perform the following kind of calculation: $$x_{(k+1)}=Ax_{(k)}$$ However, When the transition matrix $A$ is defined in the ...
0
votes
1answer
811 views

Markov Chain for system with failure and repair

I am solving the below problem, and facing a few doubts regarding my approach. Consider a system with two components. We observe the state of the system every hour. A given component operating at ...
1
vote
1answer
1k views

Use multi-state modelling (MSM) to predict individual transition probabilities in R?

I am using R package msm to model multi-state transitions in the sample dataset cav: ...
0
votes
1answer
141 views

transition matrix for urn model

There are slides regrading to urn model I have two questions if a Species A dies and a Species A is born, the original text says the probability is 0.4*0.4, but since a Species A has died , only ...
1
vote
1answer
175 views

Randomly generating transition probabilities for Markov chains

I'm trying to simulate a person moving through a household using a Markov chain. Each state would be a room in the house. The issue I'm running into is that I have no existing data telling me what a ...
1
vote
0answers
124 views

'Lumpable states' analysis for a large transition matrix

I have a large transition matrix, whereby I calculate n-step state distribution results for n=1..10, and then merge states of interest for each ...
4
votes
2answers
353 views

Transition probabilities - Markov chains

I have a homogeneous Markov chain with transition matrix I want to compute $P(Y_1 = 1| Y_2=2)$ where $Y_t, t=1,2$ is the observation at time $t$ and $Y_0=3$. I tried with Bayes' rule, so $$P(Y_1 = 1|...
2
votes
1answer
1k views

Significance testing for Markov chain transition probabilities

The question How can I calculate p values for individual transitions in a Markov chain? I want to test the null hypothesis that the probability of entering state $B$ from previous state $A$ is less ...
1
vote
1answer
72 views

What can I do with NA values in my second-order Markov chain?

I have states A, B, C, I have developed both a 1st and 2nd Order Markov Chain for them. Each state represents a status that an individual can be in, and the transitions represents the probability of ...
1
vote
1answer
420 views

Why is it necessary to fix a matrix diagonal and after this calculate the exponential to assess transition probabilities?

I'm learning markov chains in order to compute estimations of transition probabilities, and I found an example of the estimator construction for continuous time markov chains: http://www.rinfinance....
0
votes
1answer
255 views

Transition probability matrix rows not summing to 1

I'm implementing a method of sampling from a CTMC from here. I'm trying to calculate the transition probability matrix but the rows are not summing to 1, except at 0. I've diagonalised my rate ...
1
vote
0answers
287 views

Are transition matrices and markov chains considered a part of time series analysis

Is the topic of Markov Process and Transition Matrices considered a part of time series analysis? Transition matrix has the intuition of being a time based process because the transition probabilities ...