Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

Filter by
Sorted by
Tagged with
0 votes
0 answers
32 views

A recurrent Markov Chain implies its k-step version is also recurrent?

I am curious about whether a Markov Chain $X_n$ is recurrent implies that for any $k > 0$, $X_{kn}$ is also recurrent. Here are my observations. If $X_n$ is transient, $X_{kn}$ must be transient by ...
qqhgsjah8221's user avatar
8 votes
6 answers
2k views

Infinite dice roll probability

The following is an interview question: Two players A and B play a game rolling a fair die. If A rolls a 1, they immediately reroll, and if the reroll is less than 4 then A wins. Otherwise, B rolls. ...
Ria's user avatar
  • 501
4 votes
3 answers
307 views

Deriving the Distribution of Markov Chain Times

I am interested in learning how to derive the probability distributions for the Time to Absorption in Markov Chains (Discrete and Continuous). In the past, I have usually done one of the following: ...
Uk rain troll's user avatar
1 vote
0 answers
82 views

About paper "Deep Unsupervised Learning using Nonequilibrium Thermodynamics"

Here is the paper link related to the question from my title. In Appendix B, it computes the entropy of $p(X^T)$ and says "By design, the cross entropy to $\pi(x^t)$ is constant under our ...
user405729's user avatar
1 vote
1 answer
58 views

Help with Gambler's ruin problem

Players A and B each have $10 at the beginning of a game in which each player bets at each play, and the game continues until one player is broke. Suppose that Player A wins on a single bet with ...
waterr's user avatar
  • 11
1 vote
0 answers
23 views

Is it possible to Discretize a Continuous Time Markov Chain?

In a Continuous Time Markov Chain (CTMC), the following properties are said to hold: Discrete (Embedded Jump Process): $$P_{ij} = \frac{q_{ij}}{\sum_{i} q_{ij}}$$ $$q_{ij} = \lim_{{h \to 0}} \frac{...
Uk rain troll's user avatar
1 vote
0 answers
37 views

Is a sub Markov chain also a Markov chain?

Let us assume $A \rightarrow B \rightarrow C \rightarrow D$ is a markov chain. Can we also state that $A \rightarrow C \rightarrow D$ is also a Markov chain? It intuitively feels right. Can anyone ...
Bhutum Banerjee's user avatar
0 votes
0 answers
46 views

Probability that team A will win overall match

Team A and Team B are competing in a sports game and the score is currently tied at 10-10. The first team to win by a margin or two will win the tournament. Team A has 65% chance of winning each point ...
Ria's user avatar
  • 501
1 vote
0 answers
19 views

Exponential families as families of limite distributions of Markov processes

An exponential family verifies a maximum entropy property: each density is the maximum entropy density given the expectation of its sufficient statistic. On the other hand, from my understanding, the ...
Chevallier's user avatar
1 vote
0 answers
15 views

Question about the mean first passage time

A homogeneous Markov chain $\{X_n\}_{n\in\mathbb N}$ with discrete state space $\mathcal{S}$. Consider the minimum number of steps to visit $k\in \mathcal{S},$ $$\tau_{k}:=\text{min} \left\{n\ge 1:\, ...
user553010's user avatar
1 vote
1 answer
44 views

Markov Chain and deterministic function

Here is a problem I am trying to solve: Consider a sequence of IID random variables $Y_1,Y_2,Y_3,...$ with values in $E$ and let the function $\varphi: E^2 \rightarrow E$ define the corresponding ...
HellBoy's user avatar
  • 123
1 vote
0 answers
20 views

Normalization for time series comparison

I have a time series Markov Switching model, which is estimated in about 15 different versions. One or two of the time series had to be normalized in order to converge. That is 1-2 out of 15. My ...
David K's user avatar
  • 31
1 vote
1 answer
39 views

Random walk on set of integers

im trying to find the matrix and the graph of this markov chain, let $(Y_i)$ a sequence of iid RV in $\mathbb{Z}$ and identically distributed, let $Y_0$ be independent of $(Y_i)$ -idk why they said ...
HellBoy's user avatar
  • 123
0 votes
0 answers
42 views

How can I obtain the distribution of the number of customers in a non-starionary MM1 queue given an interval partition of stationary transitions?

I am considering this transient solution for the probability mass function over the number of "customers" in an MM1 queue: $$ p(k; t, i, \lambda, \mu) =\\ \exp \left( - (\lambda + \mu) t \...
Galen's user avatar
  • 8,185
0 votes
0 answers
25 views

Metropolis sampling with stochastic estimation of component of probability density

Consider the probability distribution \begin{align} p(x) = \frac{1}{Z} x^2 e^{-x^2 / 2} \end{align} where $x \in \mathbb{R}$ and $Z$ is a normalization factor so that $\int_{-\infty}^{\infty} dx \, p(...
David Albandea's user avatar
2 votes
0 answers
56 views

Proving Monotonic Decrease of Kullback-Leibler Divergence in Iterative Method for Stationary Distribution Estimation

Introduction Consider a well-behaved Markov chain with desirable properties (irreducible, aperiodic, positive recurrent), characterized by a transition matrix $P$ and a stationary distribution $\pi$. ...
Chia's user avatar
  • 43
0 votes
0 answers
8 views

Waiting/jumping times of non homogeneous process on a state chain (almost Markovian)

When considering a chain of states from 0 to N in continuous time with constant up and down going transition rates, the jumping or waiting times are exponentially distributed. Now consider the ...
Nicouh's user avatar
  • 1
0 votes
0 answers
20 views

Estimating transition probabilities and their ranges

I have a system with multiple states (N) that can transition from one state to another at every discrete time increment, or stay in the same one. I want to obtain a good estimate of the transition ...
VDC's user avatar
  • 101
0 votes
0 answers
15 views

Define Multi-Rate Network Model state space and transition rates between the states

Assume there's a network with audio and video call classes. Audio call requires 1 channel (b1) and video call classes requires different channels based on the video call type (V_low=2(b2), V_med=3(b3),...
Ajantha's user avatar
0 votes
0 answers
55 views

Conflicting results from convergence measures for MCMC

I have a Gibbs sampling algorithm, for which I would like to estimate burn-in time. The model isn't hugely complex, and I run sampling for 1000 iterations. One approach I took was tracking the running ...
Blahblahblacksheep's user avatar
0 votes
0 answers
35 views

The discretization of a continuous markov chain, is still a markov chain?

So I'm reading the Guihenneuc-Jouyaux & Robert paper from 1998 about discretization of Markov Chains in which they claim that a naive discretization of a continuous Markov chain is not usually a ...
nico's user avatar
  • 1
2 votes
0 answers
50 views

Unbiased estimate of log-likelihood of Markov bridge

Note: I have cross-posted this question to MathSE. I have the following problem I am trying to solve. I have a parametric family of "transition" distributions $p_\theta(x_{i+1}\mid x_i)$ and ...
Daniel Robert-Nicoud's user avatar
2 votes
2 answers
179 views

How to understand the definition of Markov Chain $P(X_{n+1}\in B\mid \mathcal{F}_n)=p(X_n,B)$?

The definition of Markov Chain in Durrett (Probability: Theory and Examples, 2019, Section 5.2) is: $$P(X_{n+1}\in B\mid \mathcal{F}_n)=p(X_n,B), $$ where $p$ is the Markov transition kernel ...
dodo's user avatar
  • 185
0 votes
0 answers
21 views

Question about mean hitting times

A homogeneous Markov chain $\{X_n\}_{n\in\mathbb N}$ with discrete state space $\mathcal{S}$. Set $$\tau_{k}:=\text{inf} \left\{n\ge 0:\, X_n=k \right\}.$$ where $\tau_{k}$ is defined to be $+\infty$,...
Kevin's user avatar
  • 21
0 votes
0 answers
40 views

Qustion about transient states (Continuation)

Let $$f_{kk}^{(n)}:=\Pr(X_n=k,X_v\ne k,1\le v\le n-1\mid X_0=k),~n\in \mathbb Z^+ .$$ Attributing to the comments of @Zhanxiong , I have added the other two cases to the Case 1. Case 1. Is there a ...
Kevin's user avatar
  • 21
1 vote
1 answer
71 views

Question about transient states

Is there a discrete time-homogeneous Markov chain $(X_n)_{n \geq 0}$ in which one transient state $i$ satisfies $\sum_{n=1}^{\infty}nf^{(n)}_{ii}<\infty ?$ where $$f_{ii}^{(n)}:=\Pr(X_n=i,X_v\ne i,...
Kevin's user avatar
  • 21
0 votes
2 answers
175 views

How are Markov chains memoryless when they have memory of size 1?

Markov chains are typically described as memoryless as the next state depends only on the current state but not any of the past states. But wouldn't true memorylessness mean that the next state does ...
csstudent1418's user avatar
1 vote
0 answers
80 views

The conditional expectation in Gambler’s ruin

Consider the random walk $X_0, X_1, X_2, \ldots$ on state space $S=\{0,1,\ldots,N\}$ with absorbing states $A=\{0,N\}$, and with $P(i,i+1)=p$ and $P(i,i-1)=q$ for all $i \in S \setminus A$, where $p+q=...
user250236's user avatar
0 votes
0 answers
11 views

Propagating uncertainty in a Markov chain with absorbing states

Consider a Markov model with two absorbing states $a$ and $e$ and three transient states, with an associated transition probability matrix like so: $$\begin{pmatrix} 1 & 0 & 0 & 0 & 0 \...
DRG's user avatar
  • 313
1 vote
0 answers
47 views

Markov chain data processing inequality

For a Markov chain $X \rightarrow Y \rightarrow Z$, we have the following data processing inequality: $I(Y;X) \ge I(Z;X)$. Now for the Markov chain, $(W,X) \rightarrow Y \rightarrow Z$, can we prove ...
Bhutum Banerjee's user avatar
0 votes
0 answers
19 views

What is state transition probability in an MDP? A matrix or a 3-D tensor?

I find the following written in places: $P$ is a state transition probability matrix, $P_{ss'}^{a} = P[S_{t+1} = s' | S_t = s, A_t = a]$ (notes by david silver - slide 24 and, sutton and barto) How is ...
figs_and_nuts's user avatar
1 vote
0 answers
131 views

$\mathbf{P}(S_{n(i)}=i \mid S_1=i),\mathbf{P}(S_{n(i)+1}=i\mid S_1=i),......$ are zeros?

$\left\{\xi_{n}\right\}_{n\in\ \mathbb{N}_{+}}$ is a sequence of independently and identically distributed random variables, each taking a finite number of integer values.$\mathbf{E}(\xi_1)\ne 0,$ For ...
Kevin's user avatar
  • 21
0 votes
0 answers
29 views

Proving that the average number of arrival events as $\lambda t$ given the inter-arrival duration are i.i.d. Exp($\lambda$) random variables

I'm trying to prove a common result for the Poisson process but I'm stuck. Given $T_i$ are i.i.d. $Exp(\lambda)$ random variables (where $\lambda$ is the rate) that represent the duration of arrival ...
Khai Yi Chin's user avatar
3 votes
1 answer
63 views

Using both discrete and continuous moves in Metropolis-Hastings

I want to sample a continuous distribution $f$ using the Metropolis-Hastings algorithm. Can I define my transition kernel as being sometimes discrete and sometimes continuous as long as I use the ...
siliz4's user avatar
  • 33
0 votes
1 answer
38 views

Monty Hall Permutation over undefined rounds

This is just for curiosity's sake, but I found this problem, and I don’t know how to solve it. This is not the regular Monty Hall Problem! You're a participant in yet another version of Let's Make a ...
Evan B's user avatar
  • 1
0 votes
0 answers
48 views

How to do a Markov process that generates "loopable" sequences?

I have a data set of sequential events that I could train a higher order Markov model on so that it generates sequences of events that are similar to the input set. The properties of the generated ...
Evgeniy Berezovsky's user avatar
1 vote
0 answers
20 views

Why do we need the Markovian property in Markov chains Monte Carlo?

Adaptive Markov Chains Monte Carlo (MCMC), unlike traditional MCMC methods that rely on fixed proposal distributions, dynamically adjust their proposal distribution based on the information gathered ...
user avatar
0 votes
0 answers
48 views

Adaptive Metropolis For Multidimensional Parameter

Hi recently I want to implement the adaptive Metropolis algorithm. However I dont know how to deal with multidimensional parameters. The normal step of the adaptive MCMC is to update the covariance ...
stander Qiu's user avatar
1 vote
0 answers
37 views

Sampling from a Gaussian mixture (toy example ) using MALA

Say I want to sample from a Gaussian mixture $$\pi=\sum_{i=1}^3w_i\mathcal N(x_i,\sigma_i^2I_2)\tag1$$ where he support of the 3 distributions are "effectively separated"; e.g. $w_1=.1$, $...
0xbadf00d's user avatar
  • 241
3 votes
0 answers
42 views

Is a stable VAR model geometrically ergodic

If we have a VAR process: $\begin{align} \mathbf{y}_t = A_1\mathbf{y}_{t-1} + \dots + A_d\mathbf{y}_{t-d} + \boldsymbol{\epsilon}_t, \quad t \in \mathbb{Z} \end{align}$ With the stability condition ...
Dylan Dijk's user avatar
5 votes
1 answer
220 views

Is the stationary distribution in a Markov chain just an average or will this probability distribution actually be reached?

So I know that a connected Markov chain has a stationary distribution $\pi$ that satisfies $$\lim_{t \rightarrow \infty} a_t = \pi,$$ where $a_t$ is the average probability distribution at time $t$. ...
Elena's user avatar
  • 51
2 votes
0 answers
20 views

Simulate SDE without error

Let $d,k\in\mathbb N$; $\sigma\in C^1(\mathbb R^d,\mathbb R^{d\times k})$ be Lipschitz and $\Sigma:=\sigma\sigma^\ast$; $(W_t)_{t\ge0}$ be a $k$-dimensional Brownian motion; $\lambda$ denote the ...
0xbadf00d's user avatar
  • 241
0 votes
0 answers
217 views

Simulating a Markov Chain by Hand

As a learning exercise, I am trying to learn how to fit and simulate from Continuous Time Markov Chains. Suppose I have a Stochastic Process that can assume 3 States: S1, S2 or S3. Lets say I have ...
stats_noob's user avatar
  • 8,502
0 votes
1 answer
48 views

About the requirement of iids to form a martingale

Most problems about martingales start with an assertion similar to the following one: Suppose that $X_n$ are iid such that $\mathbb{E}[|X_n|]<\infty$ and $\mathbb{E}[X_n]=0$. Then $S_n = \sum_i^n ...
user3141592's user avatar
0 votes
1 answer
1k views

Computing the Probability of travelling in the rain without an umbrella

An Individual possesses a total of $n$ umbrellas that she uses in going from her home to her office, and vice versa. If she is at home (at the office) at the beginning (end) of a day and it is ...
Win_odd Dhamnekar's user avatar
0 votes
0 answers
82 views

Why do birth death processes follow detailed balance equations?

I am following a lecture series on Markov Chains from MIT OCW over here. The lecture (and the notes) state that for birth death processes, the detailed balance equations hold. $$\pi_{i}P_{ij} = \pi_{j}...
desert_ranger's user avatar
3 votes
1 answer
99 views

Number of configurations in which you can roll 2 6's in a row

A colleague at work gave me a problem to think about. If you throw $n$ dice, how many configurations are there where you get at least 2 6's in a row? I worked a lot with Pascal's triangle and in the ...
Francesco's user avatar
0 votes
0 answers
30 views

Combination of a discrete and a continuous Markov Chain in a MCMC

Recently, I've been questioning myself on the possibility of combining a discrete update and a continuous update on a single MCMC. Stephens (2000) in Algorithm 3.2 runs the process for a fixed amount ...
nico's user avatar
  • 1
0 votes
0 answers
22 views

R codes for Bayesian Credible Intervals and p-values for a Additive Bayesian Network model

I am using Additive Bayesian Network (ABN) to model the relationship between cereal crop production and climate variables and want to generate credible intervals and p-values for my model coefficients....
Kofi Willie's user avatar
0 votes
0 answers
20 views

Non-reversible MCMC based on diffusion dynamics

Given that the current sample location is $x\in[0,1)^d$, I would like to take the next sample $y$ as $$y=x+b(x)\Delta t+\sigma(x)\sqrt{\Delta t}\xi\;;\;\;\;\xi\sim\mathcal N_{0,\:I_d}\tag1$$ for some ...
0xbadf00d's user avatar
  • 241

1
2 3 4 5
25