Questions tagged [markov-process]
A stochastic process with the property that the future is conditionally independent of the past, given the present.
1,156
questions
0
votes
0answers
27 views
How to find the n-step matrix more efficiently?
Is there any clever way to find the n-step matrix of a chain? I have the following transition matrix
However, $p^{(n)}_{1,2}$ I spent a lot of time trying to find a way of recurrence along the paths ...
0
votes
0answers
16 views
Are invariant and stationary distribution the same thing?
I am reading a material about Markov chains and in it the author works on the Markov chains part discrete the invariant distribution of the process. However, when addressing the part of continuous ...
0
votes
0answers
15 views
Birth and death process and Counting processes - What is the relationship?
Well, I started to study the birth and death process and can I say that a counting process is an example of a pure birth process or that the birth and death process is an example of a counting process?...
0
votes
0answers
10 views
Markov process of order one a Martingale?
I have two questions, and I am very confused about the concepts
Can a Markov process of order one also be a Martingale?
Is any Markov process of order one also a Martingale?
can anyone help me solve ...
0
votes
0answers
20 views
Likelihood ratio test for Markov orders - What succession of tests?
I want to estimate the Markov order of a binary sequence. For that I calculated transition matrices and the log likelihoods for the orders of interest 0,1,2 and 3. Essentially the one with the highest ...
0
votes
0answers
9 views
Calculating marginal distribution of Markov process
i am studying markov processes and we have an example of a VAR process. i am trying to understand how to look at the marginal distribution so i can find a gaussian distribution for $Y_t$ which equates ...
0
votes
0answers
7 views
Queuing theory. Modification in the M/M/1 model
In a queue of type M/M/1 when making the following modification: when there are 3 customers in the system (2 in the queue and 1 being served) if another one arrives he will leave and never come back. ...
0
votes
0answers
9 views
Periodic Markov chain in 3D state space
Does someone have an example for a discrete time (time-homogeneous) Markov chain in a three-dimensional state space that is characterized by a transition matrix resulting in periodic behaviour?
I am ...
1
vote
1answer
22 views
Markov chain and mutual information equality
Consider a distribution $P(X,Y,Z)$ and a Markov chain $Z-Z'$. Does the following equality hold in general?
$$
I(X;Y \mid Z) = I(X;Y \mid Z,Z')
$$
0
votes
0answers
14 views
Continuous time Markov process
I have to solve a problem that part of his solution goes through the following context.
Consider three balls distributed in two urns. Suppose the following process is repeated indefinitely. Let's ...
0
votes
0answers
32 views
Are these Markov Chains? $Z_i \sim Bernoulli(p)$
Question:
Suppose we have a sequence of independent and identically distributed
random variables, $Z_i$ for $i = 0,1,2,3,...$ such that for $p \in
> (0,1)$
$$ P(Z_i= z) = \begin{cases} p&\...
1
vote
0answers
18 views
Find average clients per hour for a Markov process
I'm stuck in a problem on Markov jump process:
A gas station receives cars at a rate of 20 vehicles per hour, the station has only one gas pump. If the pump is empty, it receives one client, if a new ...
0
votes
0answers
8 views
Change in Shannon Entropy in Markov Chain Process
What are the known results on the change in Shannon entropy $\Delta H_{k} = H(\vec{p}_{k}) - H(\vec{p}_{k-1})$ of the $k$-th step in a process governed by a finite state discrete time Markov chain ...
0
votes
0answers
27 views
Are stationary markov chains iid random variables?
Let $\{X_t\}_{t=1}^{\infty}$ be a Markov Chain.
An initial marginal distribution $\pi^T$ for a markov chain is a stationary distribution if $\pi^TP = \pi^T$.
My understanding of this is that if the ...
1
vote
0answers
20 views
Markov chains in nature
It is a well known result that any Markov Chain with a regular transition matrix admits a limiting distribution. Any arbitrary transition matrix can be converted into a regular transition matrix by ...
2
votes
2answers
168 views
Let $X_t$ be a solution of a SDE. Does the set $\{X_t \in \{p\}\}$ has null measure?
This question was previously posted on https://math.stackexchange.com/questions/3981156/let-x-t-be-a-solution-of-a-sde-does-the-set-x-t-in-p-has-null-meas.
I think this question is easy. However, I ...
1
vote
0answers
11 views
What is lag time in the Markov chain?
I'm pretty new in the Markov modelling. I want to know what is lag time and how does it effect the transition matrix trajectory? I have a transition matrix shown below:
$$
T_{ij} =
\begin{bmatrix}
0....
0
votes
0answers
17 views
How to generate a realization from a transition matrix?
Consider a Markov chain of 4 states described by the transition matrix,
$$
T_{ij} =
\begin{bmatrix}
0.40 & 0.56 & 0.03 & 0.01\\
0.45 & 0.51 & 0.04 & 0.00\\
0.25 & 0.25 &...
0
votes
1answer
33 views
Limiting behaviour of Markov Chains
Is the following "vague" statement correct, and if so are there a good reference out there which formally work this out?
For a given initial distribution $\vec{x}$, any finite space, ...
1
vote
1answer
37 views
Can a Markov Chain have a limiting distribution and more than 1 stationary distribution?
Can a Discrete-Time Markov Chain have a limiting distribution and more than 1 stationary distribution?
0
votes
1answer
45 views
Metropolis-Hastings with non-centered Proposal
I am trying to draw samples from the Laplace distribution $\pi^* = \text{exp}(-|\theta|)$, using Metropolis Hastings algorithm with a noncentered proposal, meaning that regular Metropolis wont work.. ...
0
votes
0answers
20 views
How to approach Model and policy in reinforcement learning?
Hey I am currently taking Stanford cs243 reinforcement learning course in Youtube to learn reinforcement learning in that I understand that policy is something like a function which get a specific ...
0
votes
0answers
6 views
Does shift-invariance of a measure in ergodic theory imply this?
I am a little confused and would appreciate some elaboration on this. Let $\mu$ be a measure on $\mathbb{R}$ equipped with the Borel $\sigma$-algebra $\mathcal{B}(\mathbb{R})$. Then the translation ...
0
votes
0answers
23 views
Invariance of a Markov process to constant time-varying additions
I am quite confident about this, but do have my doubts, as I have failed to find proof in the literature in the past hour; so would very much like some clarification. My understanding is that if the ...
2
votes
1answer
29 views
Weighted and Probability Graph
I have a simple markov chain with A, B and C states. For each state I have a probability and beyond that, a value. So, for each state transition I have two informations: the probability of the ...
0
votes
0answers
14 views
R: Multinomial logistic regression with Panel Data
Suppose you have panel data (many observations for each person) and you have a discrete ordered outcome for worker status (entry level, assistant, manager, exit labor force) that you want to model ...
2
votes
1answer
32 views
Existence of limit distribution
I know that in a Markov chain if it is aperiodic and irreducible, the stationary distribution coincides with the limit distribution. But is there anything that guarantees me the existence of the limit ...
0
votes
0answers
8 views
Sampling from extreme value distribution for arbitrary periods
If I have a yearly sample of the extreme value from a distribution $x = (x_1, x_2, \ldots, x_n)$, enough to fit to an (extreme value) distribution $F(x\,|\, \mu, \sigma, \xi)$, I can sample values $...
0
votes
0answers
16 views
Choosing a model for input: categorised, weighted sequence, output: binary variable
What would be an appropriate model for predicting a binary target variable, given a weighted sequence?
Sequences will be reasonably short, typically between ~ 1 and 5 elements. I have in the order of ...
0
votes
1answer
34 views
How do I find Schwartz criterion (or Bayesian Information Criterion) for these three models?
I have to find the schwarz criterion for each of the models in this maths question using RStudio but I don't know where to start. I know I need to find the free parameters but don't know how to find ...
6
votes
1answer
96 views
In reinforcement learning, what is the correct definition of “value function”?
This is a follow up to: In reinforcement learning, what is the correct mathematical definition of the discounted reward?
I discovered that there seems to exist an extremely large and disparate ...
0
votes
0answers
11 views
Why in Markov process only the present determines the future? What about the added knowledge of probability!
I just want to understand Markov process on a deeper level. Here the rule is $P(Xt+1=xt+1| Xt=xt, ..., X1=x1) = P(Xt+1=xt+1| Xt=xt)$
But what confuses me is that those neglected events would be ...
2
votes
2answers
169 views
Probability of doing a specific Path in a Markov Chain
My problem is the following: I have this graph, representing a Markov Chain:
For example, if I am in state 1, the probability of going in state 2 or 4 is $\frac{1}{2}$. So I'm saying that the ...
0
votes
0answers
12 views
Compute Tennis Match Winning Probabilities with Limited Predictors
I'm trying to predict tennis match outcomes in R using only match scores from previous opponents to predict winners/losers for future matches. The match scores, however, are only as granular as the ...
0
votes
0answers
14 views
Applications of continuous time markov chains
Can anyone name real life applications of continuous time Markov chains? such as birth-death process or poisson point processes.
0
votes
0answers
17 views
Infinitesimal generator
I have been studying continuous time markov chains through Dobrow's book. Everything went fine until the author introduced the concept of infinitesimal generator, which he refers to as $\textbf{Q}$. ...
0
votes
0answers
25 views
Show that $\{(X_{n},X_{n+1})\}_{n\geq 0}$ is a Markov Chain where $\{X_{n}\}_{n\geq 0}$ is a Markov Chain
Show that $\{(X_{n},X_{n+1})\}_{n\geq 0}$ is a Markov Chain where $\{X_{n}\}_{n\geq 0}$ is a Markov Chain.
Remark: We know that $\mathbb{P}(A|\emptyset)$ is undefined, I am right?
This fact is ...
0
votes
0answers
33 views
Generating correlated discrete random variables
Suppose that we have $q_t \in \{-1, 1\}$ where $\mathbb{P}(q_t = -1) = \mathbb{P}(q_t = 1) = \frac{1}{2}$. Further, assume that
\begin{align}
Cor\left( q_t, q_{t-k} \right) =
\begin{cases}
...
1
vote
1answer
23 views
Steady state distribution overdetermined linear system
I am trying to find the steady state distribution of the transition matrix given by the discrete and time-homogeneous Markov Chain with state space $S \in \{0,1,2,3,4\}$. The transition matrix is the ...
0
votes
0answers
11 views
Limiting distribution of $M/M/2/5$ queue with two heterogenous servers
A queue with a total capacity of $5$ customers has $2$ servers who serve at rates $\mu_1 = 1$ customer/hour and $\mu_2 = 2$ customers/hour respectively. Service times are exponentially distributed. ...
0
votes
0answers
20 views
2-dimensional random walk [duplicate]
I have understood that for p=1/2 1-dimensional random walk is recurrent but I have no idea on how to approach this ,in case of 2-dimensional random walk
1
vote
1answer
67 views
is this a time homogeneous markov chain
Let $S=\{1,2\}$, $\omega_1, \dots, \omega_n, \dots$ are i.i.d discrete random variable taking values in $S$, consider $f_1, f_2 :\mathbb R_{+}\to \mathbb R_{+}$ as
$f_1(x)=\begin{cases}
\...
0
votes
0answers
36 views
proof of Strong Markov property check
Statement
We say $\Phi$ has the strong Markov Property if for any initial distribution $\mu$, any real-valued bounded measurable function $h$ on $\Omega$, and any stopping time $\zeta$,
$$
\mathsf{E}_{...
0
votes
1answer
37 views
Markov transition matrix row sums to 1
I am trying to learn a little bit of Markov Chains through Dobrow's "Introduction to Stochastic Processes with R", but i am struggling with the following:
The entries of every Markov ...
6
votes
2answers
68 views
Markov chain ( Absorption)
I have just started learning Markov chain and I am clueless about how to solve this problem
A man rolls a boulder up a 40 meter-high hill. Each minute, with probability 1/3 he manages to roll the ...
6
votes
3answers
412 views
Probability after n steps
I have been studying markov chains for my Introductory Stochastic Processes exam, but i am struggling with the following problem:
Question: Consider a matrix with state space $S=\{1,2,3\}$ and the ...
0
votes
0answers
18 views
Convergence of Normalised Sum of IID Random Variables
I have a Markov chain X that starts from the stationary distribution. Let define $S_n = X_1 + \cdots + X_n.$, where $X_i$ is the state of the Markov chain. Let's have 3 states. I wanted to prove the ...
8
votes
1answer
169 views
MDP and Sate Value Finding?
I have a complex MDP (I think) as follows. anyone can describe me simply how the value for state $V(A)^*$ is find?
First Update: really for this solved question I need a canonical answer, step by ...
1
vote
1answer
41 views
How to find the equilibrium distribution of a discrete time Markov Chain
I want to find the all the equilibrium distribution of this markov chain
By letting w1= α, where 0<= α <=1, I got
w2= (7/15)α
w3 = (2/3)α
w4= (3/5)α
But I am not sure if its correct.
Also, does ...
4
votes
1answer
122 views
Probability of hitting state i
I have just started learning Markov chain and I have no idea how to solve this question
An ant walks along the edges of a cube, starting from the vertex marked 0. Upon reaching a vertex, the ant ...
