Questions tagged [markov-process]
A stochastic process with the property that the future is conditionally independent of the past, given the present.
1,236
questions
0
votes
0
answers
23
views
Applying First order stochastic dominance to Markov Matrices [closed]
I'm interested in seeing if there is a concept like first order stochastic dominance but for Markov Matrices. The reason why is because I want to know if there is a way for us to define preference for ...
- 892
0
votes
0
answers
19
views
Is the decomposition of a transitive kernel well-defined? [closed]
In the paper enclosed below, the authors write:
Suppose that the transition kernel, for some function (p(x, y)), is expressed as
$$
P(x, d y)=p(x, y) d y+r(x) \delta_x(d y),
$$
where $p(x, x)=0, \...
- 151
0
votes
0
answers
17
views
Understanding the definition of $n$-th iterate of a transition kernel
I recently came across the definition of the transition kernel for a continuous state space, which is defined recursively as follows:
\begin{aligned} & P^{(1)}(x, A)=P(x, A) \\ & P^{(n)}(x, A)...
- 151
1
vote
1
answer
14
views
R MSM package: how to flag incident and prevalent disease?
I am trying to create a 2-state Markov model (state 1 to 2 and 2 to 1) with the msm package in R. hpv_state is 2 when infected, 1 when not infected, and 999 when ...
- 11
9
votes
2
answers
362
views
Sums of exponentials joint probability
If we have that: $\tau_i \overset{\text{independent}}{\sim}
\exp(\lambda_i)$, for $i=1,2,3,...,n$, where $\lambda_i\neq \lambda_j, \forall i\neq j$ then I would like to find a general form for the ...
- 930
0
votes
0
answers
14
views
HMMs "difficulty" compared to a Markov model
Given an HMM, it is easy to compute the best approximating $n$-gram model over the observations. For example, for $N=1$, we have $p(w_i|w_{i-1}) = \sum_{s_i,s_{i-1}}p(w_i,s_i|w_{i-1},s_{i-1})=\sum_{...
- 277
0
votes
0
answers
20
views
Estimating markov transition matrix using total elevator "ups" and "downs" by floor
I have data on elevator presses and I am hoping to use them to estimate a Markov transition matrix, so I can ultimately estimate how frequently people go to different floors.
For each floor from 1-4, ...
- 156
1
vote
0
answers
14
views
Estimating Markov Chain Probabilities with Limited Data
Suppose I have some data on transitions between states of a Discrete Time Markov Chain. Let's say that transitions between some events are observed more frequently from others. For example, in a 3 ...
user414238
9
votes
2
answers
543
views
Markov Chains with Changing Number of States
I have seen these kinds of Discrete State Markov Chains before (Continuous Time or Discrete Time):
Homogeneous (Probability Transition Matrix is constant)
Non-Homogeneous (Probability Transition ...
- 91
3
votes
0
answers
64
views
+100
How to tune the unadjusted Langevin algorithm?
I want to start investigating the (unadjusted) simulation of the Langevin process $${\rm d}X_t=b(X_t){\rm d}t+\sigma{\rm d}W_t,$$ where $$b:=\frac{\sigma^2}2\nabla\ln p.$$ I don't want to simulate ...
- 213
0
votes
0
answers
35
views
Why reverse diffusion process is not a gaussian distribution?
The forward diffusion process, which goes from x_t to x_{t+1} is Gaussian, which is very reasonable as we go the next state by adding random gaussian noise. However, I do not understand why the ...
- 185
0
votes
0
answers
60
views
Sum of powers (geometric series) of state transition matrix
I am working discrete time Markov chain analysis for some large state transition graph. I want to find the rewards/cost to reach from the init state to the terminal/accepting states.
I have the state ...
- 101
0
votes
0
answers
7
views
Which variable is best suited for edge weights when computing graph algorithms instead of relative risks?
I am currently trying to develop graph data. Which variable is best suited for edge weights when computing graph algorithms?
Relative risk
Relative Risk: Many networks in my field use relative risks ...
- 61
0
votes
0
answers
13
views
Forward-Backward Algorithm for Autoregressive HMMs
I am currently studying HMMs, and covered the Forward-Backward Algorithms as well as the smoothing and filtering process. Recently, we were posed a question on Autoregressive HMMs which I've been ...
- 83
0
votes
1
answer
115
views
msm package: Mutlti state model initial value in 'vmmin' is not finite
I am new to msm package and markov models. I have a randomized trial dataset with readings from three time points: baseline, at 1 year, and at 2 year. I am trying to calculate annual transition ...
- 1
4
votes
1
answer
160
views
Can MCMC sample any probability distributions?
I have three fundamental questions related to MCMC. I would appreciate the help on any one of those.
The most fundamental question in MCMC field, which I can't find a reference, is: Can MCMC generate ...
- 53
1
vote
1
answer
48
views
Show that the total variation distance of the Metropolis kernel to its proposal kernel is equal to the rejection probability
Furthermore, let
$(E,\mathcal E,\lambda)$ be a $\sigma$-finite measure space;
$Q$ be a Markov kernel on $(E,\mathcal E)$ with density $q$ with respect to $\lambda$;
$\mu$ be a probability measure on $...
- 213
0
votes
0
answers
16
views
How can we compare the "performance" of different Markov chain Monte Carlo algorithms?
How can we judge the performance a Markov chain Monte Carlo (MCMC) algorithm? I guess we could consider one of the following:
The variance of $X_t$ for a given $t\in I$;
The asymptotic variance of $(...
- 213
0
votes
0
answers
10
views
What is the effect of sampling rate on parameter estimation when fitting a markov state model to timeseries data?
Let us say that I have some timeseries data, which can be described by a markov state model. And the time series has been sampled every $\Delta t$ time units. The sampling rate ($1/\Delta t$) must ...
- 1
0
votes
0
answers
15
views
How to test Markovian property in a financial time series?
I want to build a Markov Chain model for a financial time series to determine transition probabilities from one state to another. The underlying assumption is that series obeys the Markovian property. ...
- 487
-1
votes
0
answers
33
views
Help with Gambler's ruin problem, can't solve abstraction [duplicate]
I'm having difficulty solving this exercise. When I assume that p=0.4 and player A's fortune is 99 dollars and B's fortune is 1 dollar, I can find that the probability of player A losing to player B ...
- 1
1
vote
1
answer
55
views
Quesiton about markov chain period of transient
I got a quick question I got a markov chain with this trans matrix $$\begin{pmatrix}1&0\\1/2&1/2\end{pmatrix}$$
And I got 2 states right [0,1] right.
So I know state 0 has a period 1 and is ...
0
votes
0
answers
37
views
A recurrent Markov Chain implies its k-step version is also recurrent?
I am curious about whether a Markov Chain $X_n$ is recurrent implies that for any $k > 0$, $X_{kn}$ is also recurrent.
Here are my observations. If $X_n$ is transient, $X_{kn}$ must be transient by ...
- 21
8
votes
6
answers
2k
views
Infinite dice roll probability
The following is an interview question:
Two players A and B play a game rolling a fair die. If A rolls a 1, they immediately reroll, and if the reroll is less than 4 then A wins. Otherwise, B rolls. ...
- 491
4
votes
3
answers
441
views
Deriving the Distribution of Markov Chain Times
I am interested in learning how to derive the probability distributions for the Time to Absorption in Markov Chains (Discrete and Continuous).
In the past, I have usually done one of the following:
...
1
vote
0
answers
120
views
About paper "Deep Unsupervised Learning using Nonequilibrium Thermodynamics"
Here is the paper link related to the question from my title.
In Appendix B, it computes the entropy of $p(X^T)$ and says "By design, the cross entropy to $\pi(x^t)$ is constant under our ...
- 11
1
vote
1
answer
75
views
Help with Gambler's ruin problem
Players A and B each have $10 at the beginning of a game in which each player bets at each play, and the game continues until one player is broke. Suppose that Player A wins on a single bet with ...
- 41
1
vote
0
answers
34
views
Is it possible to Discretize a Continuous Time Markov Chain?
In a Continuous Time Markov Chain (CTMC), the following properties are said to hold:
Discrete (Embedded Jump Process):
$$P_{ij} = \frac{q_{ij}}{\sum_{i} q_{ij}}$$
$$q_{ij} = \lim_{{h \to 0}} \frac{...
1
vote
0
answers
59
views
Is a sub Markov chain also a Markov chain?
Let us assume $A \rightarrow B \rightarrow C \rightarrow D$ is a markov chain. Can we also state that $A \rightarrow C \rightarrow D$ is also a Markov chain? It intuitively feels right. Can anyone ...
0
votes
0
answers
115
views
Probability that team A will win overall match
Team A and Team B are competing in a sports game and the score is currently tied at 10-10. The first team to win by a margin or two will win the tournament. Team A has 65% chance of winning each point ...
- 491
1
vote
0
answers
26
views
Exponential families as families of limite distributions of Markov processes
An exponential family verifies a maximum entropy property: each density is the maximum entropy density given the expectation of its sufficient statistic.
On the other hand, from my understanding, the ...
- 151
1
vote
0
answers
20
views
Question about the mean first passage time
A homogeneous Markov chain $\{X_n\}_{n\in\mathbb N}$ with discrete state space $\mathcal{S}$.
Consider the minimum number of steps to visit $k\in \mathcal{S},$
$$\tau_{k}:=\text{min} \left\{n\ge 1:\, ...
- 11
1
vote
1
answer
47
views
Markov Chain and deterministic function
Here is a problem I am trying to solve:
Consider a sequence of IID random variables $Y_1,Y_2,Y_3,...$ with values in $E$ and let the function $\varphi: E^2 \rightarrow E$ define the corresponding ...
- 123
1
vote
0
answers
29
views
Normalization for time series comparison
I have a time series Markov Switching model, which is estimated in about 15 different versions. One or two of the time series had to be normalized in order to converge. That is 1-2 out of 15. My ...
- 31
1
vote
1
answer
42
views
Random walk on set of integers
im trying to find the matrix and the graph of this markov chain, let $(Y_i)$ a sequence of iid RV in $\mathbb{Z}$ and identically distributed, let $Y_0$ be independent of $(Y_i)$ -idk why they said ...
- 123
0
votes
0
answers
44
views
How can I obtain the distribution of the number of customers in a non-starionary MM1 queue given an interval partition of stationary transitions?
I am considering this transient solution for the probability mass function over the number of "customers" in an MM1 queue:
$$ p(k; t, i, \lambda, \mu) =\\ \exp \left( - (\lambda + \mu) t \...
- 9,522
0
votes
0
answers
26
views
Metropolis sampling with stochastic estimation of component of probability density
Consider the probability distribution
\begin{align}
p(x) = \frac{1}{Z} x^2 e^{-x^2 / 2}
\end{align}
where $x \in \mathbb{R}$ and $Z$ is a normalization factor so that $\int_{-\infty}^{\infty} dx \, p(...
- 133
2
votes
0
answers
81
views
Proving Monotonic Decrease of Kullback-Leibler Divergence in Iterative Method for Stationary Distribution Estimation
Introduction
Consider a well-behaved Markov chain with desirable properties (irreducible, aperiodic, positive recurrent), characterized by a transition matrix $P$ and a stationary distribution $\pi$. ...
- 43
0
votes
0
answers
10
views
Waiting/jumping times of non homogeneous process on a state chain (almost Markovian)
When considering a chain of states from 0 to N in continuous time with constant up and down going transition rates, the jumping or waiting times are exponentially distributed. Now consider the ...
- 1
0
votes
0
answers
24
views
Estimating transition probabilities and their ranges
I have a system with multiple states (N) that can transition from one state to another at every discrete time increment, or stay in the same one. I want to obtain a good estimate of the transition ...
- 101
0
votes
0
answers
18
views
Define Multi-Rate Network Model state space and transition rates between the states
Assume there's a network with audio and video call classes. Audio call requires 1 channel (b1) and video call classes requires different channels based on the video call type (V_low=2(b2), V_med=3(b3),...
- 1
0
votes
0
answers
82
views
Conflicting results from convergence measures for MCMC
I have a Gibbs sampling algorithm, for which I would like to estimate burn-in time. The model isn't hugely complex, and I run sampling for 1000 iterations.
One approach I took was tracking the running ...
0
votes
0
answers
41
views
The discretization of a continuous markov chain, is still a markov chain?
So I'm reading the Guihenneuc-Jouyaux & Robert paper from 1998 about discretization of Markov Chains in which they claim that a naive discretization of a continuous Markov chain is not usually a ...
- 1
2
votes
0
answers
52
views
Unbiased estimate of log-likelihood of Markov bridge
Note: I have cross-posted this question to MathSE.
I have the following problem I am trying to solve. I have a parametric family of "transition" distributions $p_\theta(x_{i+1}\mid x_i)$ and ...
2
votes
2
answers
197
views
How to understand the definition of Markov Chain $P(X_{n+1}\in B\mid \mathcal{F}_n)=p(X_n,B)$?
The definition of Markov Chain in Durrett (Probability: Theory and Examples, 2019, Section 5.2) is:
$$P(X_{n+1}\in B\mid \mathcal{F}_n)=p(X_n,B), $$
where $p$ is the Markov transition kernel ...
- 185
0
votes
0
answers
22
views
Question about mean hitting times
A homogeneous Markov chain $\{X_n\}_{n\in\mathbb N}$ with discrete state space $\mathcal{S}$.
Set
$$\tau_{k}:=\text{inf} \left\{n\ge 0:\, X_n=k \right\}.$$
where $\tau_{k}$ is defined to be $+\infty$,...
- 31
0
votes
0
answers
40
views
Qustion about transient states (Continuation)
Let $$f_{kk}^{(n)}:=\Pr(X_n=k,X_v\ne k,1\le v\le n-1\mid X_0=k),~n\in \mathbb Z^+ .$$
Attributing to the comments of @Zhanxiong , I have added the other two cases to the Case 1.
Case 1.
Is there a ...
- 31
1
vote
1
answer
76
views
Question about transient states
Is there a discrete time-homogeneous Markov chain $(X_n)_{n \geq 0}$ in which one transient state $i$ satisfies $\sum_{n=1}^{\infty}nf^{(n)}_{ii}<\infty ?$
where $$f_{ii}^{(n)}:=\Pr(X_n=i,X_v\ne i,...
- 31
0
votes
2
answers
320
views
How are Markov chains memoryless when they have memory of size 1?
Markov chains are typically described as memoryless as the next state depends only on the current state but not any of the past states.
But wouldn't true memorylessness mean that the next state does ...
- 109
1
vote
0
answers
83
views
The conditional expectation in Gambler’s ruin
Consider the random walk $X_0, X_1, X_2, \ldots$ on state space $S=\{0,1,\ldots,N\}$ with absorbing states $A=\{0,N\}$, and with $P(i,i+1)=p$ and $P(i,i-1)=q$ for all $i \in S \setminus A$, where $p+q=...
- 111