Skip to main content

Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

Filter by
Sorted by
Tagged with
0 votes
0 answers
16 views

Why reverse diffusion process is not a gaussian distribution?

The forward diffusion process, which goes from x_t to x_{t+1} is Gaussian, which is very reasonable as we go the next state by adding random gaussian noise. However, I do not understand why the ...
levitatmas's user avatar
0 votes
0 answers
43 views

Sum of powers (geometric series) of state transition matrix

I am working discrete time Markov chain analysis for some large state transition graph. I want to find the rewards/cost to reach from the init state to the terminal/accepting states. I have the state ...
JackDaniels's user avatar
0 votes
0 answers
3 views

Which variable is best suited for edge weights when computing graph algorithms instead of relative risks?

I am currently trying to develop graph data. Which variable is best suited for edge weights when computing graph algorithms? Relative risk Relative Risk: Many networks in my field use relative risks ...
user1190107's user avatar
0 votes
0 answers
10 views

Forward-Backward Algorithm for Autoregressive HMMs

I am currently studying HMMs, and covered the Forward-Backward Algorithms as well as the smoothing and filtering process. Recently, we were posed a question on Autoregressive HMMs which I've been ...
Kai's user avatar
  • 83
0 votes
1 answer
32 views

msm package: Mutlti state model initial value in 'vmmin' is not finite

I am new to msm package and markov models. I have a randomized trial dataset with readings from three time points: baseline, at 1 year, and at 2 year. I am trying to calculate annual transition ...
spri0330's user avatar
3 votes
1 answer
133 views

Can MCMC sample any probability distributions?

I have three fundamental questions related to MCMC. I would appreciate the help on any one of those. The most fundamental question in MCMC field, which I can't find a reference, is: Can MCMC generate ...
George Lu's user avatar
2 votes
1 answer
39 views

Show that the total variation distance of the Metropolis kernel to its proposal kernel is equal to the rejection probability

Furthermore, let $(E,\mathcal E,\lambda)$ be a $\sigma$-finite measure space; $Q$ be a Markov kernel on $(E,\mathcal E)$ with density $q$ with respect to $\lambda$; $\mu$ be a probability measure on $...
0xbadf00d's user avatar
  • 303
0 votes
0 answers
15 views

How can we compare the "performance" of different Markov chain Monte Carlo algorithms?

How can we judge the performance a Markov chain Monte Carlo (MCMC) algorithm? I guess we could consider one of the following: The variance of $X_t$ for a given $t\in I$; The asymptotic variance of $(...
0xbadf00d's user avatar
  • 303
0 votes
0 answers
8 views

What is the effect of sampling rate on parameter estimation when fitting a markov state model to timeseries data?

Let us say that I have some timeseries data, which can be described by a markov state model. And the time series has been sampled every $\Delta t$ time units. The sampling rate ($1/\Delta t$) must ...
ace_101's user avatar
0 votes
0 answers
7 views

How to test Markovian property in a financial time series?

I want to build a Markov Chain model for a financial time series to determine transition probabilities from one state to another. The underlying assumption is that series obeys the Markovian property. ...
Sane's user avatar
  • 353
-1 votes
0 answers
33 views

Help with Gambler's ruin problem, can't solve abstraction [duplicate]

I'm having difficulty solving this exercise. When I assume that p=0.4 and player A's fortune is 99 dollars and B's fortune is 1 dollar, I can find that the probability of player A losing to player B ...
Insomnia's user avatar
1 vote
1 answer
36 views

Quesiton about markov chain period of transient

I got a quick question I got a markov chain with this trans matrix $$\begin{pmatrix}1&0\\1/2&1/2\end{pmatrix}$$ And I got 2 states right [0,1] right. So I know state 0 has a period 1 and is ...
Fernando Martinez's user avatar
0 votes
0 answers
35 views

A recurrent Markov Chain implies its k-step version is also recurrent?

I am curious about whether a Markov Chain $X_n$ is recurrent implies that for any $k > 0$, $X_{kn}$ is also recurrent. Here are my observations. If $X_n$ is transient, $X_{kn}$ must be transient by ...
qqhgsjah8221's user avatar
8 votes
6 answers
2k views

Infinite dice roll probability

The following is an interview question: Two players A and B play a game rolling a fair die. If A rolls a 1, they immediately reroll, and if the reroll is less than 4 then A wins. Otherwise, B rolls. ...
Ria's user avatar
  • 501
4 votes
3 answers
343 views

Deriving the Distribution of Markov Chain Times

I am interested in learning how to derive the probability distributions for the Time to Absorption in Markov Chains (Discrete and Continuous). In the past, I have usually done one of the following: ...
Uk rain troll's user avatar
1 vote
0 answers
96 views

About paper "Deep Unsupervised Learning using Nonequilibrium Thermodynamics"

Here is the paper link related to the question from my title. In Appendix B, it computes the entropy of $p(X^T)$ and says "By design, the cross entropy to $\pi(x^t)$ is constant under our ...
user405729's user avatar
1 vote
1 answer
63 views

Help with Gambler's ruin problem

Players A and B each have $10 at the beginning of a game in which each player bets at each play, and the game continues until one player is broke. Suppose that Player A wins on a single bet with ...
waterr's user avatar
  • 41
1 vote
0 answers
31 views

Is it possible to Discretize a Continuous Time Markov Chain?

In a Continuous Time Markov Chain (CTMC), the following properties are said to hold: Discrete (Embedded Jump Process): $$P_{ij} = \frac{q_{ij}}{\sum_{i} q_{ij}}$$ $$q_{ij} = \lim_{{h \to 0}} \frac{...
Uk rain troll's user avatar
1 vote
0 answers
44 views

Is a sub Markov chain also a Markov chain?

Let us assume $A \rightarrow B \rightarrow C \rightarrow D$ is a markov chain. Can we also state that $A \rightarrow C \rightarrow D$ is also a Markov chain? It intuitively feels right. Can anyone ...
Bhutum Banerjee's user avatar
0 votes
0 answers
55 views

Probability that team A will win overall match

Team A and Team B are competing in a sports game and the score is currently tied at 10-10. The first team to win by a margin or two will win the tournament. Team A has 65% chance of winning each point ...
Ria's user avatar
  • 501
1 vote
0 answers
20 views

Exponential families as families of limite distributions of Markov processes

An exponential family verifies a maximum entropy property: each density is the maximum entropy density given the expectation of its sufficient statistic. On the other hand, from my understanding, the ...
Chevallier's user avatar
1 vote
0 answers
17 views

Question about the mean first passage time

A homogeneous Markov chain $\{X_n\}_{n\in\mathbb N}$ with discrete state space $\mathcal{S}$. Consider the minimum number of steps to visit $k\in \mathcal{S},$ $$\tau_{k}:=\text{min} \left\{n\ge 1:\, ...
user553010's user avatar
1 vote
1 answer
47 views

Markov Chain and deterministic function

Here is a problem I am trying to solve: Consider a sequence of IID random variables $Y_1,Y_2,Y_3,...$ with values in $E$ and let the function $\varphi: E^2 \rightarrow E$ define the corresponding ...
HellBoy's user avatar
  • 123
1 vote
0 answers
22 views

Normalization for time series comparison

I have a time series Markov Switching model, which is estimated in about 15 different versions. One or two of the time series had to be normalized in order to converge. That is 1-2 out of 15. My ...
David K's user avatar
  • 31
1 vote
1 answer
41 views

Random walk on set of integers

im trying to find the matrix and the graph of this markov chain, let $(Y_i)$ a sequence of iid RV in $\mathbb{Z}$ and identically distributed, let $Y_0$ be independent of $(Y_i)$ -idk why they said ...
HellBoy's user avatar
  • 123
0 votes
0 answers
42 views

How can I obtain the distribution of the number of customers in a non-starionary MM1 queue given an interval partition of stationary transitions?

I am considering this transient solution for the probability mass function over the number of "customers" in an MM1 queue: $$ p(k; t, i, \lambda, \mu) =\\ \exp \left( - (\lambda + \mu) t \...
Galen's user avatar
  • 8,754
0 votes
0 answers
26 views

Metropolis sampling with stochastic estimation of component of probability density

Consider the probability distribution \begin{align} p(x) = \frac{1}{Z} x^2 e^{-x^2 / 2} \end{align} where $x \in \mathbb{R}$ and $Z$ is a normalization factor so that $\int_{-\infty}^{\infty} dx \, p(...
David Albandea's user avatar
2 votes
0 answers
66 views

Proving Monotonic Decrease of Kullback-Leibler Divergence in Iterative Method for Stationary Distribution Estimation

Introduction Consider a well-behaved Markov chain with desirable properties (irreducible, aperiodic, positive recurrent), characterized by a transition matrix $P$ and a stationary distribution $\pi$. ...
Chia's user avatar
  • 43
0 votes
0 answers
8 views

Waiting/jumping times of non homogeneous process on a state chain (almost Markovian)

When considering a chain of states from 0 to N in continuous time with constant up and down going transition rates, the jumping or waiting times are exponentially distributed. Now consider the ...
Nicouh's user avatar
  • 1
0 votes
0 answers
21 views

Estimating transition probabilities and their ranges

I have a system with multiple states (N) that can transition from one state to another at every discrete time increment, or stay in the same one. I want to obtain a good estimate of the transition ...
VDC's user avatar
  • 101
0 votes
0 answers
16 views

Define Multi-Rate Network Model state space and transition rates between the states

Assume there's a network with audio and video call classes. Audio call requires 1 channel (b1) and video call classes requires different channels based on the video call type (V_low=2(b2), V_med=3(b3),...
Ajantha's user avatar
0 votes
0 answers
65 views

Conflicting results from convergence measures for MCMC

I have a Gibbs sampling algorithm, for which I would like to estimate burn-in time. The model isn't hugely complex, and I run sampling for 1000 iterations. One approach I took was tracking the running ...
Blahblahblacksheep's user avatar
0 votes
0 answers
35 views

The discretization of a continuous markov chain, is still a markov chain?

So I'm reading the Guihenneuc-Jouyaux & Robert paper from 1998 about discretization of Markov Chains in which they claim that a naive discretization of a continuous Markov chain is not usually a ...
nico's user avatar
  • 1
2 votes
0 answers
51 views

Unbiased estimate of log-likelihood of Markov bridge

Note: I have cross-posted this question to MathSE. I have the following problem I am trying to solve. I have a parametric family of "transition" distributions $p_\theta(x_{i+1}\mid x_i)$ and ...
Daniel Robert-Nicoud's user avatar
2 votes
2 answers
189 views

How to understand the definition of Markov Chain $P(X_{n+1}\in B\mid \mathcal{F}_n)=p(X_n,B)$?

The definition of Markov Chain in Durrett (Probability: Theory and Examples, 2019, Section 5.2) is: $$P(X_{n+1}\in B\mid \mathcal{F}_n)=p(X_n,B), $$ where $p$ is the Markov transition kernel ...
dodo's user avatar
  • 185
0 votes
0 answers
21 views

Question about mean hitting times

A homogeneous Markov chain $\{X_n\}_{n\in\mathbb N}$ with discrete state space $\mathcal{S}$. Set $$\tau_{k}:=\text{inf} \left\{n\ge 0:\, X_n=k \right\}.$$ where $\tau_{k}$ is defined to be $+\infty$,...
Kevin's user avatar
  • 31
0 votes
0 answers
40 views

Qustion about transient states (Continuation)

Let $$f_{kk}^{(n)}:=\Pr(X_n=k,X_v\ne k,1\le v\le n-1\mid X_0=k),~n\in \mathbb Z^+ .$$ Attributing to the comments of @Zhanxiong , I have added the other two cases to the Case 1. Case 1. Is there a ...
Kevin's user avatar
  • 31
1 vote
1 answer
72 views

Question about transient states

Is there a discrete time-homogeneous Markov chain $(X_n)_{n \geq 0}$ in which one transient state $i$ satisfies $\sum_{n=1}^{\infty}nf^{(n)}_{ii}<\infty ?$ where $$f_{ii}^{(n)}:=\Pr(X_n=i,X_v\ne i,...
Kevin's user avatar
  • 31
0 votes
2 answers
223 views

How are Markov chains memoryless when they have memory of size 1?

Markov chains are typically described as memoryless as the next state depends only on the current state but not any of the past states. But wouldn't true memorylessness mean that the next state does ...
csstudent1418's user avatar
1 vote
0 answers
80 views

The conditional expectation in Gambler’s ruin

Consider the random walk $X_0, X_1, X_2, \ldots$ on state space $S=\{0,1,\ldots,N\}$ with absorbing states $A=\{0,N\}$, and with $P(i,i+1)=p$ and $P(i,i-1)=q$ for all $i \in S \setminus A$, where $p+q=...
user250236's user avatar
0 votes
0 answers
12 views

Propagating uncertainty in a Markov chain with absorbing states

Consider a Markov model with two absorbing states $a$ and $e$ and three transient states, with an associated transition probability matrix like so: $$\begin{pmatrix} 1 & 0 & 0 & 0 & 0 \...
DRG's user avatar
  • 313
1 vote
0 answers
51 views

Markov chain data processing inequality

For a Markov chain $X \rightarrow Y \rightarrow Z$, we have the following data processing inequality: $I(Y;X) \ge I(Z;X)$. Now for the Markov chain, $(W,X) \rightarrow Y \rightarrow Z$, can we prove ...
Bhutum Banerjee's user avatar
0 votes
0 answers
22 views

What is state transition probability in an MDP? A matrix or a 3-D tensor?

I find the following written in places: $P$ is a state transition probability matrix, $P_{ss'}^{a} = P[S_{t+1} = s' | S_t = s, A_t = a]$ (notes by david silver - slide 24 and, sutton and barto) How is ...
figs_and_nuts's user avatar
1 vote
0 answers
133 views

$\mathbf{P}(S_{n(i)}=i \mid S_1=i),\mathbf{P}(S_{n(i)+1}=i\mid S_1=i),......$ are zeros?

$\left\{\xi_{n}\right\}_{n\in\ \mathbb{N}_{+}}$ is a sequence of independently and identically distributed random variables, each taking a finite number of integer values.$\mathbf{E}(\xi_1)\ne 0,$ For ...
Kevin's user avatar
  • 31
0 votes
0 answers
29 views

Proving that the average number of arrival events as $\lambda t$ given the inter-arrival duration are i.i.d. Exp($\lambda$) random variables

I'm trying to prove a common result for the Poisson process but I'm stuck. Given $T_i$ are i.i.d. $Exp(\lambda)$ random variables (where $\lambda$ is the rate) that represent the duration of arrival ...
Khai Yi Chin's user avatar
3 votes
1 answer
69 views

Using both discrete and continuous moves in Metropolis-Hastings

I want to sample a continuous distribution $f$ using the Metropolis-Hastings algorithm. Can I define my transition kernel as being sometimes discrete and sometimes continuous as long as I use the ...
siliz4's user avatar
  • 33
1 vote
1 answer
52 views

Monty Hall Permutation over undefined rounds

This is just for curiosity's sake, but I found this problem, and I don’t know how to solve it. This is not the regular Monty Hall Problem! You're a participant in yet another version of Let's Make a ...
Evan B's user avatar
  • 11
0 votes
0 answers
49 views

How to do a Markov process that generates "loopable" sequences?

I have a data set of sequential events that I could train a higher order Markov model on so that it generates sequences of events that are similar to the input set. The properties of the generated ...
Evgeniy Berezovsky's user avatar
1 vote
0 answers
25 views

Why do we need the Markovian property in Markov chains Monte Carlo?

Adaptive Markov Chains Monte Carlo (MCMC), unlike traditional MCMC methods that rely on fixed proposal distributions, dynamically adjust their proposal distribution based on the information gathered ...
user avatar
0 votes
0 answers
56 views

Adaptive Metropolis For Multidimensional Parameter

Hi recently I want to implement the adaptive Metropolis algorithm. However I dont know how to deal with multidimensional parameters. The normal step of the adaptive MCMC is to update the covariance ...
stander Qiu's user avatar

1
2 3 4 5
25