Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

Is it possible to eye a reversible Markov Chain by it's transition matrix?

I was wondering if it's possible. Many of the other Markov-Chain properties can be (somewhat) easily eyed from the transition matrix. (e.g. irreducible - if there's an absorbing state (1 in the ...
1
vote
1answer
12 views

Calculating the expected number of visits to a state by a DTMC

Suppose we have a DTMC $X$ : $\{X_n : n = 0, 1,2,\dots\}$, a transition probability matrix $P$, and state space $S = \{1,2,3\}$. Suppose I want to calculate the expected amount of times we visit state ...
1
vote
0answers
20 views

Using Forward Backward algorithm to find posterior probability of all possible states

I understand that Viterbi finds the most probable sequence of states. However, I want the probability of all possible sequences of states. I understand that FB algorithm can be used to find the ...
0
votes
0answers
12 views

Why is Markov Property useful for modeling systems?

I came across this question in my data science interview practice, and am not sure how to answer. Markov models are based on the assumption of Markov property, so it is useful for modeling Markov ...
3
votes
1answer
16 views

Conditional probabilities with Discrete Time Markov Chain non-perishable inventory

Suppose $\{X_n : n =0,1,2,\dots \}$ is a DTMC that represents the inventory level at the end of day $n$. We have inventory policy $(2,4)$, i.e., if $X_n < 2$, we order enough units to have ...
4
votes
1answer
102 views

Is gamma actually an efficient way to weigh future rewards in reinforcement learning?

Typically the discounted sum of rewards is defined as follows: G_t = Sum(gamma ** n * reward_t...) But this means that rewards are worth exponentially less with ...
0
votes
0answers
23 views

MCMC beginner question at an example chain plot: Do I need more steps? How much burn-in do I need, if I can tell already?

I am using the emcee python library to fit a model to data via MCMC. Below an example plot for the chain of one of my parameters. Here I ran 1000 steps with 100 walkers. Now I have two beginner ...
0
votes
0answers
9 views

Fitting the transition matrix of a Markov chain given the series of state

Almost all examples I can found on Markov chain and python are giving the transition matrix as known. Is there any library that can fit the transition matrix from the series of state observations?
1
vote
0answers
35 views

Prove the joint distribution of an AR(1) process is Multivariate Gaussian

I'm struggling a bit with the proof of this if anyone can help! I keep ending up going in circles with the conditional probabilities and I don't know what are the right steps to take. For an $AR(1)$ ...
2
votes
1answer
21 views

To compute the class of states for the given transition probability matrix

I have been given the following transition probability matrix of a markov chain: $P = \begin{pmatrix} \frac{3}{4}{} & 0 & \frac{1}{4} &0 \\ \frac{1}{2} & 0 & 0 & \frac{1}{2}\\ ...
1
vote
1answer
41 views

A Markov chain {Xn, n ≥ 0} with states 1, 2,3 has the transition probability matrix with an initial distribution (1/2,0,1/2), what is P(X1=3|X2=1)

A Markov chain {Xn, n ≥ 0} with states 1, 2,3 has the transition probability matrix P \begin{bmatrix}0&0.4&0.6\\1&0&0\\0.3&0.3&0.4\end{bmatrix} with an initial distribution A (...
0
votes
1answer
27 views

Whats exactly deterministic and non deterministic in deterministic and nondeterministic MDP policies?

Consider below Markov Decision Process: Blue hexagons are states and orange circles are actions. I have rather simple confusion. What will be nature of deterministic and non deterministic MDPs? This ...
1
vote
1answer
54 views

Equivalence of two state Markov chain and sampling via geometric distribution

Let $\mathcal T = \{1,2,\ldots,T\}$ denote the set of points in time, $S = \{0,1\}$ the state space, $X = (X_t)_{t \in \mathcal T} \in S^\mathcal T$ a time series, $\alpha = \mathbb P(X_{t+1} = 0 \mid ...
0
votes
0answers
11 views

Is it possible to occupy two states at the same time in Markov process/chain?

As in the topic - is it possible for an agent in a system of Markov's fashion to move into two states at once?
0
votes
0answers
28 views

How to find the n-step matrix more efficiently?

Is there any clever way to find the n-step matrix of a chain? I have the following transition matrix However, $p^{(n)}_{1,2}$ I spent a lot of time trying to find a way of recurrence along the paths ...
0
votes
0answers
17 views

Are invariant and stationary distribution the same thing?

I am reading a material about Markov chains and in it the author works on the Markov chains part discrete the invariant distribution of the process. However, when addressing the part of continuous ...
0
votes
0answers
18 views

Birth and death process and Counting processes - What is the relationship?

Well, I started to study the birth and death process and can I say that a counting process is an example of a pure birth process or that the birth and death process is an example of a counting process?...
0
votes
0answers
12 views

Markov process of order one a Martingale?

I have two questions, and I am very confused about the concepts Can a Markov process of order one also be a Martingale? Is any Markov process of order one also a Martingale? can anyone help me solve ...
0
votes
0answers
22 views

Likelihood ratio test for Markov orders - What succession of tests?

I want to estimate the Markov order of a binary sequence. For that I calculated transition matrices and the log likelihoods for the orders of interest 0,1,2 and 3. Essentially the one with the highest ...
0
votes
0answers
9 views

Calculating marginal distribution of Markov process

i am studying markov processes and we have an example of a VAR process. i am trying to understand how to look at the marginal distribution so i can find a gaussian distribution for $Y_t$ which equates ...
0
votes
0answers
7 views

Queuing theory. Modification in the M/M/1 model

In a queue of type M/M/1 when making the following modification: when there are 3 customers in the system (2 in the queue and 1 being served) if another one arrives he will leave and never come back. ...
0
votes
0answers
9 views

Periodic Markov chain in 3D state space

Does someone have an example for a discrete time (time-homogeneous) Markov chain in a three-dimensional state space that is characterized by a transition matrix resulting in periodic behaviour? I am ...
1
vote
1answer
31 views

Markov chain and mutual information equality

Consider a distribution $P(X,Y,Z)$ and a Markov chain $Z-Z'$. Does the following equality hold in general? $$ I(X;Y \mid Z) = I(X;Y \mid Z,Z') $$
0
votes
0answers
14 views

Continuous time Markov process

I have to solve a problem that part of his solution goes through the following context. Consider three balls distributed in two urns. Suppose the following process is repeated indefinitely. Let's ...
0
votes
0answers
35 views

Are these Markov Chains? $Z_i \sim Bernoulli(p)$

Question: Suppose we have a sequence of independent and identically distributed random variables, $Z_i$ for $i = 0,1,2,3,...$ such that for $p \in > (0,1)$ $$ P(Z_i= z) = \begin{cases} p&\...
1
vote
0answers
18 views

Find average clients per hour for a Markov process

I'm stuck in a problem on Markov jump process: A gas station receives cars at a rate of 20 vehicles per hour, the station has only one gas pump. If the pump is empty, it receives one client, if a new ...
0
votes
0answers
12 views

Change in Shannon Entropy in Markov Chain Process

What are the known results on the change in Shannon entropy $\Delta H_{k} = H(\vec{p}_{k}) - H(\vec{p}_{k-1})$ of the $k$-th step in a process governed by a finite state discrete time Markov chain ...
0
votes
0answers
32 views

Are stationary markov chains iid random variables?

Let $\{X_t\}_{t=1}^{\infty}$ be a Markov Chain. An initial marginal distribution $\pi^T$ for a markov chain is a stationary distribution if $\pi^TP = \pi^T$. My understanding of this is that if the ...
1
vote
0answers
20 views

Markov chains in nature

It is a well known result that any Markov Chain with a regular transition matrix admits a limiting distribution. Any arbitrary transition matrix can be converted into a regular transition matrix by ...
2
votes
2answers
221 views

Let $X_t$ be a solution of a SDE. Does the set $\{X_t \in \{p\}\}$ has null measure?

This question was previously posted on https://math.stackexchange.com/questions/3981156/let-x-t-be-a-solution-of-a-sde-does-the-set-x-t-in-p-has-null-meas. I think this question is easy. However, I ...
1
vote
0answers
12 views

What is lag time in the Markov chain?

I'm pretty new in the Markov modelling. I want to know what is lag time and how does it effect the transition matrix trajectory? I have a transition matrix shown below: $$ T_{ij} = \begin{bmatrix} 0....
0
votes
0answers
17 views

How to generate a realization from a transition matrix?

Consider a Markov chain of 4 states described by the transition matrix, $$ T_{ij} = \begin{bmatrix} 0.40 & 0.56 & 0.03 & 0.01\\ 0.45 & 0.51 & 0.04 & 0.00\\ 0.25 & 0.25 &...
0
votes
1answer
36 views

Limiting behaviour of Markov Chains

Is the following "vague" statement correct, and if so are there a good reference out there which formally work this out? For a given initial distribution $\vec{x}$, any finite space, ...
1
vote
1answer
41 views

Can a Markov Chain have a limiting distribution and more than 1 stationary distribution?

Can a Discrete-Time Markov Chain have a limiting distribution and more than 1 stationary distribution?
0
votes
1answer
48 views

Metropolis-Hastings with non-centered Proposal

I am trying to draw samples from the Laplace distribution $\pi^* = \text{exp}(-|\theta|)$, using Metropolis Hastings algorithm with a noncentered proposal, meaning that regular Metropolis wont work.. ...
0
votes
0answers
21 views

How to approach Model and policy in reinforcement learning?

Hey I am currently taking Stanford cs243 reinforcement learning course in Youtube to learn reinforcement learning in that I understand that policy is something like a function which get a specific ...
0
votes
0answers
7 views

Does shift-invariance of a measure in ergodic theory imply this?

I am a little confused and would appreciate some elaboration on this. Let $\mu$ be a measure on $\mathbb{R}$ equipped with the Borel $\sigma$-algebra $\mathcal{B}(\mathbb{R})$. Then the translation ...
0
votes
0answers
23 views

Invariance of a Markov process to constant time-varying additions

I am quite confident about this, but do have my doubts, as I have failed to find proof in the literature in the past hour; so would very much like some clarification. My understanding is that if the ...
2
votes
1answer
29 views

Weighted and Probability Graph

I have a simple markov chain with A, B and C states. For each state I have a probability and beyond that, a value. So, for each state transition I have two informations: the probability of the ...
0
votes
0answers
16 views

R: Multinomial logistic regression with Panel Data

Suppose you have panel data (many observations for each person) and you have a discrete ordered outcome for worker status (entry level, assistant, manager, exit labor force) that you want to model ...
2
votes
1answer
35 views

Existence of limit distribution

I know that in a Markov chain if it is aperiodic and irreducible, the stationary distribution coincides with the limit distribution. But is there anything that guarantees me the existence of the limit ...
0
votes
0answers
9 views

Sampling from extreme value distribution for arbitrary periods

If I have a yearly sample of the extreme value from a distribution $x = (x_1, x_2, \ldots, x_n)$, enough to fit to an (extreme value) distribution $F(x\,|\, \mu, \sigma, \xi)$, I can sample values $...
0
votes
0answers
17 views

Choosing a model for input: categorised, weighted sequence, output: binary variable

What would be an appropriate model for predicting a binary target variable, given a weighted sequence? Sequences will be reasonably short, typically between ~ 1 and 5 elements. I have in the order of ...
0
votes
1answer
37 views

How do I find Schwartz criterion (or Bayesian Information Criterion) for these three models?

I have to find the schwarz criterion for each of the models in this maths question using RStudio but I don't know where to start. I know I need to find the free parameters but don't know how to find ...
6
votes
1answer
104 views

In reinforcement learning, what is the correct definition of “value function”?

This is a follow up to: In reinforcement learning, what is the correct mathematical definition of the discounted reward? I discovered that there seems to exist an extremely large and disparate ...
0
votes
0answers
11 views

Why in Markov process only the present determines the future? What about the added knowledge of probability!

I just want to understand Markov process on a deeper level. Here the rule is $P(Xt+1=xt+1| Xt=xt, ..., X1=x1) = P(Xt+1=xt+1| Xt=xt)$ But what confuses me is that those neglected events would be ...
2
votes
2answers
175 views

Probability of doing a specific Path in a Markov Chain

My problem is the following: I have this graph, representing a Markov Chain: For example, if I am in state 1, the probability of going in state 2 or 4 is $\frac{1}{2}$. So I'm saying that the ...
0
votes
0answers
12 views

Compute Tennis Match Winning Probabilities with Limited Predictors

I'm trying to predict tennis match outcomes in R using only match scores from previous opponents to predict winners/losers for future matches. The match scores, however, are only as granular as the ...
0
votes
0answers
14 views

Applications of continuous time markov chains

Can anyone name real life applications of continuous time Markov chains? such as birth-death process or poisson point processes.
0
votes
0answers
18 views

Infinitesimal generator

I have been studying continuous time markov chains through Dobrow's book. Everything went fine until the author introduced the concept of infinitesimal generator, which he refers to as $\textbf{Q}$. ...

1
2 3 4 5
24