Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

Filter by
Sorted by
Tagged with
0
votes
0answers
27 views

How to find the n-step matrix more efficiently?

Is there any clever way to find the n-step matrix of a chain? I have the following transition matrix However, $p^{(n)}_{1,2}$ I spent a lot of time trying to find a way of recurrence along the paths ...
0
votes
0answers
16 views

Are invariant and stationary distribution the same thing?

I am reading a material about Markov chains and in it the author works on the Markov chains part discrete the invariant distribution of the process. However, when addressing the part of continuous ...
0
votes
0answers
15 views

Birth and death process and Counting processes - What is the relationship?

Well, I started to study the birth and death process and can I say that a counting process is an example of a pure birth process or that the birth and death process is an example of a counting process?...
0
votes
0answers
10 views

Markov process of order one a Martingale?

I have two questions, and I am very confused about the concepts Can a Markov process of order one also be a Martingale? Is any Markov process of order one also a Martingale? can anyone help me solve ...
0
votes
0answers
20 views

Likelihood ratio test for Markov orders - What succession of tests?

I want to estimate the Markov order of a binary sequence. For that I calculated transition matrices and the log likelihoods for the orders of interest 0,1,2 and 3. Essentially the one with the highest ...
0
votes
0answers
9 views

Calculating marginal distribution of Markov process

i am studying markov processes and we have an example of a VAR process. i am trying to understand how to look at the marginal distribution so i can find a gaussian distribution for $Y_t$ which equates ...
0
votes
0answers
7 views

Queuing theory. Modification in the M/M/1 model

In a queue of type M/M/1 when making the following modification: when there are 3 customers in the system (2 in the queue and 1 being served) if another one arrives he will leave and never come back. ...
0
votes
0answers
9 views

Periodic Markov chain in 3D state space

Does someone have an example for a discrete time (time-homogeneous) Markov chain in a three-dimensional state space that is characterized by a transition matrix resulting in periodic behaviour? I am ...
1
vote
1answer
22 views

Markov chain and mutual information equality

Consider a distribution $P(X,Y,Z)$ and a Markov chain $Z-Z'$. Does the following equality hold in general? $$ I(X;Y \mid Z) = I(X;Y \mid Z,Z') $$
0
votes
0answers
14 views

Continuous time Markov process

I have to solve a problem that part of his solution goes through the following context. Consider three balls distributed in two urns. Suppose the following process is repeated indefinitely. Let's ...
0
votes
0answers
32 views

Are these Markov Chains? $Z_i \sim Bernoulli(p)$

Question: Suppose we have a sequence of independent and identically distributed random variables, $Z_i$ for $i = 0,1,2,3,...$ such that for $p \in > (0,1)$ $$ P(Z_i= z) = \begin{cases} p&\...
1
vote
0answers
18 views

Find average clients per hour for a Markov process

I'm stuck in a problem on Markov jump process: A gas station receives cars at a rate of 20 vehicles per hour, the station has only one gas pump. If the pump is empty, it receives one client, if a new ...
0
votes
0answers
8 views

Change in Shannon Entropy in Markov Chain Process

What are the known results on the change in Shannon entropy $\Delta H_{k} = H(\vec{p}_{k}) - H(\vec{p}_{k-1})$ of the $k$-th step in a process governed by a finite state discrete time Markov chain ...
0
votes
0answers
27 views

Are stationary markov chains iid random variables?

Let $\{X_t\}_{t=1}^{\infty}$ be a Markov Chain. An initial marginal distribution $\pi^T$ for a markov chain is a stationary distribution if $\pi^TP = \pi^T$. My understanding of this is that if the ...
1
vote
0answers
20 views

Markov chains in nature

It is a well known result that any Markov Chain with a regular transition matrix admits a limiting distribution. Any arbitrary transition matrix can be converted into a regular transition matrix by ...
2
votes
2answers
168 views

Let $X_t$ be a solution of a SDE. Does the set $\{X_t \in \{p\}\}$ has null measure?

This question was previously posted on https://math.stackexchange.com/questions/3981156/let-x-t-be-a-solution-of-a-sde-does-the-set-x-t-in-p-has-null-meas. I think this question is easy. However, I ...
1
vote
0answers
11 views

What is lag time in the Markov chain?

I'm pretty new in the Markov modelling. I want to know what is lag time and how does it effect the transition matrix trajectory? I have a transition matrix shown below: $$ T_{ij} = \begin{bmatrix} 0....
0
votes
0answers
17 views

How to generate a realization from a transition matrix?

Consider a Markov chain of 4 states described by the transition matrix, $$ T_{ij} = \begin{bmatrix} 0.40 & 0.56 & 0.03 & 0.01\\ 0.45 & 0.51 & 0.04 & 0.00\\ 0.25 & 0.25 &...
0
votes
1answer
33 views

Limiting behaviour of Markov Chains

Is the following "vague" statement correct, and if so are there a good reference out there which formally work this out? For a given initial distribution $\vec{x}$, any finite space, ...
1
vote
1answer
37 views

Can a Markov Chain have a limiting distribution and more than 1 stationary distribution?

Can a Discrete-Time Markov Chain have a limiting distribution and more than 1 stationary distribution?
0
votes
1answer
45 views

Metropolis-Hastings with non-centered Proposal

I am trying to draw samples from the Laplace distribution $\pi^* = \text{exp}(-|\theta|)$, using Metropolis Hastings algorithm with a noncentered proposal, meaning that regular Metropolis wont work.. ...
0
votes
0answers
20 views

How to approach Model and policy in reinforcement learning?

Hey I am currently taking Stanford cs243 reinforcement learning course in Youtube to learn reinforcement learning in that I understand that policy is something like a function which get a specific ...
0
votes
0answers
6 views

Does shift-invariance of a measure in ergodic theory imply this?

I am a little confused and would appreciate some elaboration on this. Let $\mu$ be a measure on $\mathbb{R}$ equipped with the Borel $\sigma$-algebra $\mathcal{B}(\mathbb{R})$. Then the translation ...
0
votes
0answers
23 views

Invariance of a Markov process to constant time-varying additions

I am quite confident about this, but do have my doubts, as I have failed to find proof in the literature in the past hour; so would very much like some clarification. My understanding is that if the ...
2
votes
1answer
29 views

Weighted and Probability Graph

I have a simple markov chain with A, B and C states. For each state I have a probability and beyond that, a value. So, for each state transition I have two informations: the probability of the ...
0
votes
0answers
14 views

R: Multinomial logistic regression with Panel Data

Suppose you have panel data (many observations for each person) and you have a discrete ordered outcome for worker status (entry level, assistant, manager, exit labor force) that you want to model ...
2
votes
1answer
32 views

Existence of limit distribution

I know that in a Markov chain if it is aperiodic and irreducible, the stationary distribution coincides with the limit distribution. But is there anything that guarantees me the existence of the limit ...
0
votes
0answers
8 views

Sampling from extreme value distribution for arbitrary periods

If I have a yearly sample of the extreme value from a distribution $x = (x_1, x_2, \ldots, x_n)$, enough to fit to an (extreme value) distribution $F(x\,|\, \mu, \sigma, \xi)$, I can sample values $...
0
votes
0answers
16 views

Choosing a model for input: categorised, weighted sequence, output: binary variable

What would be an appropriate model for predicting a binary target variable, given a weighted sequence? Sequences will be reasonably short, typically between ~ 1 and 5 elements. I have in the order of ...
0
votes
1answer
34 views

How do I find Schwartz criterion (or Bayesian Information Criterion) for these three models?

I have to find the schwarz criterion for each of the models in this maths question using RStudio but I don't know where to start. I know I need to find the free parameters but don't know how to find ...
6
votes
1answer
96 views

In reinforcement learning, what is the correct definition of “value function”?

This is a follow up to: In reinforcement learning, what is the correct mathematical definition of the discounted reward? I discovered that there seems to exist an extremely large and disparate ...
0
votes
0answers
11 views

Why in Markov process only the present determines the future? What about the added knowledge of probability!

I just want to understand Markov process on a deeper level. Here the rule is $P(Xt+1=xt+1| Xt=xt, ..., X1=x1) = P(Xt+1=xt+1| Xt=xt)$ But what confuses me is that those neglected events would be ...
2
votes
2answers
169 views

Probability of doing a specific Path in a Markov Chain

My problem is the following: I have this graph, representing a Markov Chain: For example, if I am in state 1, the probability of going in state 2 or 4 is $\frac{1}{2}$. So I'm saying that the ...
0
votes
0answers
12 views

Compute Tennis Match Winning Probabilities with Limited Predictors

I'm trying to predict tennis match outcomes in R using only match scores from previous opponents to predict winners/losers for future matches. The match scores, however, are only as granular as the ...
0
votes
0answers
14 views

Applications of continuous time markov chains

Can anyone name real life applications of continuous time Markov chains? such as birth-death process or poisson point processes.
0
votes
0answers
17 views

Infinitesimal generator

I have been studying continuous time markov chains through Dobrow's book. Everything went fine until the author introduced the concept of infinitesimal generator, which he refers to as $\textbf{Q}$. ...
0
votes
0answers
25 views

Show that $\{(X_{n},X_{n+1})\}_{n\geq 0}$ is a Markov Chain where $\{X_{n}\}_{n\geq 0}$ is a Markov Chain

Show that $\{(X_{n},X_{n+1})\}_{n\geq 0}$ is a Markov Chain where $\{X_{n}\}_{n\geq 0}$ is a Markov Chain. Remark: We know that $\mathbb{P}(A|\emptyset)$ is undefined, I am right? This fact is ...
0
votes
0answers
33 views

Generating correlated discrete random variables

Suppose that we have $q_t \in \{-1, 1\}$ where $\mathbb{P}(q_t = -1) = \mathbb{P}(q_t = 1) = \frac{1}{2}$. Further, assume that \begin{align} Cor\left( q_t, q_{t-k} \right) = \begin{cases} ...
1
vote
1answer
23 views

Steady state distribution overdetermined linear system

I am trying to find the steady state distribution of the transition matrix given by the discrete and time-homogeneous Markov Chain with state space $S \in \{0,1,2,3,4\}$. The transition matrix is the ...
0
votes
0answers
11 views

Limiting distribution of $M/M/2/5$ queue with two heterogenous servers

A queue with a total capacity of $5$ customers has $2$ servers who serve at rates $\mu_1 = 1$ customer/hour and $\mu_2 = 2$ customers/hour respectively. Service times are exponentially distributed. ...
0
votes
0answers
20 views

2-dimensional random walk [duplicate]

I have understood that for p=1/2 1-dimensional random walk is recurrent but I have no idea on how to approach this ,in case of 2-dimensional random walk
1
vote
1answer
67 views

is this a time homogeneous markov chain

Let $S=\{1,2\}$, $\omega_1, \dots, \omega_n, \dots$ are i.i.d discrete random variable taking values in $S$, consider $f_1, f_2 :\mathbb R_{+}\to \mathbb R_{+}$ as $f_1(x)=\begin{cases} \...
0
votes
0answers
36 views

proof of Strong Markov property check

Statement We say $\Phi$ has the strong Markov Property if for any initial distribution $\mu$, any real-valued bounded measurable function $h$ on $\Omega$, and any stopping time $\zeta$, $$ \mathsf{E}_{...
0
votes
1answer
37 views

Markov transition matrix row sums to 1

I am trying to learn a little bit of Markov Chains through Dobrow's "Introduction to Stochastic Processes with R", but i am struggling with the following: The entries of every Markov ...
6
votes
2answers
68 views

Markov chain ( Absorption)

I have just started learning Markov chain and I am clueless about how to solve this problem A man rolls a boulder up a 40 meter-high hill. Each minute, with probability 1/3 he manages to roll the ...
6
votes
3answers
412 views

Probability after n steps

I have been studying markov chains for my Introductory Stochastic Processes exam, but i am struggling with the following problem: Question: Consider a matrix with state space $S=\{1,2,3\}$ and the ...
0
votes
0answers
18 views

Convergence of Normalised Sum of IID Random Variables

I have a Markov chain X that starts from the stationary distribution. Let define $S_n = X_1 + \cdots + X_n.$, where $X_i$ is the state of the Markov chain. Let's have 3 states. I wanted to prove the ...
8
votes
1answer
169 views

MDP and Sate Value Finding?

I have a complex MDP (I think) as follows. anyone can describe me simply how the value for state $V(A)^*$ is find? First Update: really for this solved question I need a canonical answer, step by ...
1
vote
1answer
41 views

How to find the equilibrium distribution of a discrete time Markov Chain

I want to find the all the equilibrium distribution of this markov chain By letting w1= α, where 0<= α <=1, I got w2= (7/15)α w3 = (2/3)α w4= (3/5)α But I am not sure if its correct. Also, does ...
4
votes
1answer
122 views

Probability of hitting state i

I have just started learning Markov chain and I have no idea how to solve this question An ant walks along the edges of a cube, starting from the vertex marked 0. Upon reaching a vertex, the ant ...

1
2 3 4 5
24