Questions tagged [transition-matrix]

A transition matrix is a square matrix used to describe the transitions of a Markov chain.

Filter by
Sorted by
Tagged with
2
votes
0answers
85 views

How to interpret clusters on Markov chain time characteristics?

I have a discrete-time Markov chain. The Markov chain is aperiodic (because self-loops exist) and is irreducible. I have found the mean recurrence time (left graph) and then sorted mean recurrence ...
0
votes
0answers
30 views

Singular state transition probability matrix in David Silver's UCL Lesson 2

I'm studying David Silver's second lesson on reinforcement learning: https://www.youtube.com/watch?v=lfHX2hHRMVQ&list=PLqYmG7hTraZDM-OYHWgPebj2MfCFzFObQ&index=2 and the state transition ...
0
votes
0answers
9 views

Specifying a multi-state model with unobservable terminal state

Suppose there is a multi-state process with three states, listed below and labelled as terminal/non-terminal and observable/unobservable: Initializing: non-terminal observable Active: non-terminal ...
0
votes
0answers
11 views

Transition Matrix definition in State Space models using gestural data

I am trying to represent some gestural data (x,y,z from right hand & x,y,z from left hand) I am getting from sensors in a state space form, so as to predict the next x,y,z. Since statistics &...
0
votes
0answers
64 views

Make sense of plotting a transition matrix

I'm studying statistics and I'm trying to understand markov chain topic. I'm using the package "markovchain" in R to obtain the stationary distribution. From this transition matrix $M$: ...
0
votes
0answers
16 views

Markov chain transition Matrix for Ion channels in Python

I was working to code for a markov chain transition matrix for potassium channels. Potassium channels conists of 4 gates 1,2,3,4. the potassium ions antransition from 1->2 or 1->1, 2->1,2->2,2->3 etc. ...
0
votes
0answers
118 views

Measure the distance between two probability transition matrices

I have a probability transition matrix $P$ that contains some values very close to zero. I want to sparsify this matrix by taking the k largest values for each row and setting the others to zero. For ...
0
votes
1answer
41 views

Long run proportion of transitions in a Markov chain

Let $S$ be a set of states for a Markov chain and let $S^C$ be the remaining states. Explain the identity $$\sum_{i\in S}\sum_{j\in S^C}\pi_iP_{ij}=\sum_{i\in S^C}\sum_{j\in S}\pi_iP_{ij}$$ I know ...
0
votes
0answers
122 views

Transition probability in Markov chain using a decision tree model

I wish to find a way to calculate the transition probabilities in my Markov chain model. Let's say a customer has three products [A B C] and in this example I wish to know the transition probability ...
1
vote
1answer
124 views

Irreducible (communicating) classes [closed]

The Markov chain $(Xn; n\geq)$ has state-space $S = (0, 1, 2, . . .)$, with $p_{i,0} = \frac{1}{4}$ and $p_{i,i+1} = \frac{3}{4}$ $\forall i \geq 0$, so that the transition matrix is P =$\...
0
votes
1answer
86 views

Biased coins and Markov processes

Good day, I am attempting an optional exercise and I am finding it hard to interpret the problem in terms of matrices and vectors. Coin 1 has probability 0.4 of coming up heads, and coin 2 has ...
0
votes
0answers
99 views

Formulating a Transition matrix for Markov Process

I am dealing with a medical process which is as follows. There are 10000 Veterans who are enrolled in this study. All 10000 have medical condition called onychocryptosis which is a fancy term for ...
2
votes
1answer
243 views

What is the difference betwen a time non-homogenous Markov Chain and a non-linear Markov Chain? Example

A time non-homogenous Markov Chain is one in which the transition probabilities are not constant over time. A non-linear Markov Chain is a model that is not linear in parameters and satisfies the ...
1
vote
0answers
400 views

Irreducible Markov chain and transition matrix

We know that a matrix is irreducible if it is not similar via a permutation to a block upper triangular matrix. Is the transition matrix of a irreducible Markov chain irreducible?
1
vote
0answers
147 views

Bayesian inference of parameter governing Markov transition matrix

A 3-state Markov chain $X = \{x_i : i \in \{1, \cdots, N\}\}$ is observed, and its transition matrix $P$ is assumed to be of the form $$ \begin{pmatrix} (1-a)^2 & 2a(1-a) & a^2 \\ b(1-a) &...
1
vote
0answers
108 views

Bayesian inference of non-homogeneous Markov transition matrix

The data consists of several discrete-time Markov chains, indexed by a global time. I assume all the chains are governed by the same transition matrix, but that this can change in time. I want to ...
2
votes
0answers
261 views

A Hidden Markov model with covariates in the transition probabilities

I would like to construct a Hidden Markov model with data about online customer journeys. A well-known concept related to the customer journey literature is the sales funnel. Consumers walk through ...
0
votes
0answers
77 views

How two transition probability matrices can be used to find one variable in analysis

I'm working on a research paper for cricket analysis but I'm stuck on the point of the estimation of $\tau_{owj}$, which is defined in the appendix. I have calculated up through $\alpha_{owj}$ and $\...
0
votes
0answers
512 views

Second order markov tranision probability matrix

I tried to find the second order Markov chain of the following sequence Dat= A A B A B A A A B B A A B I tried it on "Markov chain" package in R. ...
0
votes
2answers
415 views

Markov-Chain transition probabilities for 3 variables

I am a bit confused as I need to calculate the Markov-Chain transition probabilites for 3 variables. Example data, let's assume a sequence of letters at specific and progressively-constant time steps:...
0
votes
1answer
521 views

Seeking examples of latent transition analysis with ordinal data and multiple subjects

Background A colleague of mine has asked me for help. She has a large amount of patient data involving clinical psychological measures (e.g., questions related to specific symptoms) gathered both ...
0
votes
1answer
719 views

MLE for Markov Chains - intuitive explanation

could anyone please give me intuitive explanation what does below mean ? Let say I have sequence: a, b, a, b, b, b, a, b, b, a By applying Markov Chains with MLE method in R package on below I get ...
0
votes
1answer
296 views

Irreducible Markov Chain Question

If you have an irreducible Markov chain with transition matrix $P$, and $p(j,j) > 0$ for all $j$, why are all states aperiodic?
1
vote
1answer
676 views

“Symmetric” property of stationary distribution

The above symmetric property isn't referring to the double stochastic property \begin{bmatrix} 0.2&0.8&0&0&0\\ 0.2&0.2&0.6&0&0\\ 0&0.4&0.2&0.4&0\\ 0&...
0
votes
1answer
1k views

Expected value in Markov chains

Let $\left\{X_{n}\right\}_{n\geq0}$ be a homogeneous markov chain with state space E and transition matrix P. Let $\tau$ be the first time n for which $X_{n}$ $\neq$ $X_{0}$, where $\tau=+\infty$ if $...
1
vote
0answers
61 views

How to find certain probabilities in transition matrices

A unit consists of three circuit elements in parallel; it will be in working order as long as at least one of the elements is working. The unit is examined every hour and if it is not in working order ...
0
votes
0answers
100 views

How to bet on a binary event based on the markov transition matrix, state probabilities and the odds

There is a coupon full of football matches for a given day from a bookkeeper. I have scrapped another website and i have aquired continuous history of a particular match between ...
3
votes
1answer
2k views

How to translate an adjacency matrix to a transition matrix for use in Markov cluster algorithm? [closed]

I have a matrix of size (47*47 double) that have only 0's and 1's. I want to apply the Markov clustering algorithm on this matrix, but this Method needs a transition matrix as the columns must be ...
4
votes
0answers
281 views

Find the invariant measure $\pi=(\pi_{1},\pi_{2},\pi_{3})$ for a Markov Chain with transition matrix given

Let $(X_{n})_{n\in\mathbb{N}_{0}}$ be a Markov Chain with state space $M=\left\{x_{1},x_{2},x_{3}\right\}$ and transtition matrix $$ \Pi=\left(\begin{array}{ccc}p_{1} & p_{2} & 1-p_{1}-p_{2}\\ ...
0
votes
0answers
29 views

Calculate the transition matrix of $X_{n+1}:= \sum_{i=0}^{X_{n}}\theta_{n}^{i}\:\: \mbox{mod }5.$ where $\theta_{n}^{i}\sim Bin(3,1/3)$ i.i.d

Given an i.i.d. sequence $(\theta_{n}^{i})_{n,i\in\mathbb{N}}$ with binomial distribution $\mathcal{B}_{3,\frac{1}{3}}$ we define the Markov Chain $$X_{n+1}:= \sum_{i=0}^{X_{n}}\theta_{n}^{i}\:\:\: \...
4
votes
1answer
853 views

Constructing a transition probability from Q-learning

In Reinforcement learning, learning without the need for the transition probability matrix is 'model free learning'. Instead of having the transition probabilities, we learn the q-values (state/action ...
1
vote
0answers
34 views

Markov process with conditions

I am trying to model a basketball game using play by play data. Each team has a transition matrix representing what they will do next on offense and defense in any given time remaining in the game. ...
3
votes
1answer
620 views

Finding steady-state probability of a Markov chain

Let $X_{n}$ be a Markov chain on state space $S = \{ 1,2 \dots, 23 \}$ with transition probability given by $p_{i,i+1}= p_{i,i-1} = \frac {1}{2} \ \ \forall \ 2\le i \le 22 , $ $ p_{1,2}= p_{1,23} =...
4
votes
4answers
2k views

Essential transient state in a Markov chain

Can a finite state Markov chain have essential transient state? I have found out an example for an infinite state one and I have the intuition (I may be wrong) that for a finite state space .. This ...
2
votes
0answers
313 views

The expected long run proportion of time the chain spends at $a$ , given that it starts at $c$

Consider the transition matrix: $\begin{bmatrix} \frac{1}{5} & \frac{4}{5} & 0 & 0 & 0 \\ \frac{1}{2} & \frac{1}{2} & 0 & 0 & 0 \\ \frac{1}{5} & \frac{1}{5} & \...
0
votes
0answers
125 views

Stationary Probabilities

A transition probability matrix $P$ is said to be doubly stochastic, if each column sum is 1;that is if $\displaystyle\sum_i P_{i,j} = 1$ for all j If such a chain is irreducible and have states $1,...
0
votes
0answers
287 views

When Markov Chain goes to stable?

Suppose I have a transition matrix $A$, which is positive definite. I know that this chain will goes to stable after some finite time. The question is when will this chain goes to stable? Or, after ...
7
votes
0answers
308 views

Reinforcement *Model* Learning

Classical reinforcement learning (Q- or Sarsa-Learning) can be extended with models of the environment. These models are usually transition tables that contain the probability of arriving at a ...
0
votes
1answer
688 views

Regular Transition Matrix

A transition matrix is regular if some power of the matrix contains all positive entries. [1] Its powers have all positive entries... Why isn't this matrix a Regular Transition Matrix? Reference: ...
1
vote
0answers
320 views

how to calculate transition probability matrix for n*4 data

data: https://www.dropbox.com/s/4ni96jmzn14sjl4/florep_normn4_p1.mat?dl=0 I attached n*4 data. so here each row has 4 values and each row considered as one state. I want to calculate transition ...
1
vote
1answer
397 views

Transition matrix calculation from text book “Introduction to information retrieval”

According to the example on page in the text book, the graph of documents is as follows: $1\rightarrow2, 3\rightarrow2, 2\rightarrow1, 2\rightarrow3$ with $\lambda = 0.5$ which would form an ...
2
votes
1answer
786 views

Markov chain transition matrix and state vector order?

The typical way of performing state simulations using matrices is to perform the following kind of calculation: $$x_{(k+1)}=Ax_{(k)}$$ However, When the transition matrix $A$ is defined in the ...
0
votes
1answer
978 views

Markov Chain for system with failure and repair

I am solving the below problem, and facing a few doubts regarding my approach. Consider a system with two components. We observe the state of the system every hour. A given component operating at ...
2
votes
1answer
1k views

Use multi-state modelling (MSM) to predict individual transition probabilities in R?

I am using R package msm to model multi-state transitions in the sample dataset cav: ...
0
votes
1answer
160 views

transition matrix for urn model

There are slides regrading to urn model I have two questions if a Species A dies and a Species A is born, the original text says the probability is 0.4*0.4, but since a Species A has died , only ...
1
vote
1answer
243 views

Randomly generating transition probabilities for Markov chains

I'm trying to simulate a person moving through a household using a Markov chain. Each state would be a room in the house. The issue I'm running into is that I have no existing data telling me what a ...
1
vote
0answers
134 views

'Lumpable states' analysis for a large transition matrix

I have a large transition matrix, whereby I calculate n-step state distribution results for n=1..10, and then merge states of interest for each ...
4
votes
2answers
413 views

Transition probabilities - Markov chains

I have a homogeneous Markov chain with transition matrix I want to compute $P(Y_1 = 1| Y_2=2)$ where $Y_t, t=1,2$ is the observation at time $t$ and $Y_0=3$. I tried with Bayes' rule, so $$P(Y_1 = 1|...
2
votes
1answer
2k views

Significance testing for Markov chain transition probabilities

The question How can I calculate p values for individual transitions in a Markov chain? I want to test the null hypothesis that the probability of entering state $B$ from previous state $A$ is less ...
1
vote
1answer
78 views

What can I do with NA values in my second-order Markov chain?

I have states A, B, C, I have developed both a 1st and 2nd Order Markov Chain for them. Each state represents a status that an individual can be in, and the transitions represents the probability of ...