Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

0
votes
0answers
21 views

Probabilistic user behavior markov models on web [on hold]

I am considering the following probabilistic Markov model of actions of a user on the results page of a search engine. The user examines the first result, with a probability $A$ he is satisfied with ...
0
votes
0answers
10 views

markov decision process question [on hold]

I have some question from the lecture notes, the problem is and the probability matrix is I don't know how these three matrix come out, especially what does "a" works?
0
votes
0answers
24 views

States of Markov chain and stationary distribution

Let $X$ be a Markov chain with a state space $S={\{0,1,2,... \}}$ and a transition matrix $P$ with given $p_{i,0}=\frac{i}{i+1}$ and $p_{i,i+1}=\frac{1}{i+1}$, for $i=0,1,2,...$. Find out which states ...
3
votes
1answer
62 views

Proving that given Markov chain is homogeneous. Find state space and transition matrix

Let $X_i$ be the results of a consecutive throws of a die. Let $Z_n=3(X_1^2+\cdots+X_n^2) \bmod 5$. Show that the sequence ${\{Z_n \mid n\geq1\}}$ is a homogeneous Markov Chain. Find a state space and ...
0
votes
0answers
7 views

Reinforcement learning with two goal states [closed]

I implemented my first TD off-policy method for discrete states and discrete actions (increase or decrease to last executed action) on a patient in real time. Now I am extending my problem from one ...
1
vote
0answers
24 views

Expected number of steps in Gambler's ruin game with two players

Let's say we have two players A and B. Player A has 3 coins and player B has 5 coins. If player wins the other player gives one coin. During game second player probability of loosing is $2/3$, while ...
1
vote
1answer
46 views

Why is variational Bayesian mixture model an alternative to MCMC? What are the similarities?

Why do people say that a variational Bayesian mixture model could be an alternative to MCMC for clustering? For example see the details here: https://en.wikipedia.org/wiki/Variational_Bayesian_method. ...
1
vote
0answers
17 views

Importance of the right-continuity of filtration in definition of strong Markov Property

Taking the definition from wikipedia, With $X = (X_t : t \geq 0) $ as a stochastic process on a probability space $(\Omega, \mathcal{F}, \mathbb{P})$ with natural filtration $\{ \mathcal{F}(t) \}_{t \...
0
votes
1answer
21 views

Can a state have zero periodicity? [closed]

I am getting my concepts cleared in Stochastic process. I understand the concept of periodicity. Just to make it clear, suppose there is a finite Markov chain with states $1,2,3$. Let their ...
1
vote
1answer
56 views

Why is the probability of a random walk reaching 1 (in n steps) squared greater than the probability of it reaching 2 (in n steps)?

Let $S_n$ be a simple random walk. i.e. $$ S_n = \sum_{t=1}^n X_t, $$ where ${X_t}$ are i.i.d random variables with $$ X_t = \begin{cases} +1, & \textrm{w/ probability } p \\ -1, & \...
1
vote
0answers
16 views

Convergence in total distribution distance in the Random Walk Metropolis-Hastings algorithm

I'm searching for a proof of the convergence in total distribution distance of the transition probabilities of a Markov chain generated by the Random Walk Metropolis-Hastings algorithm to its ...
0
votes
1answer
44 views

Generating very few samples from a probability distribution using MCMC?

Since MCMC converges to target only after taking very large number of steps, what if I want to have just say 10 samples from target distribution? Do I still have to generate lots of samples, and then ...
1
vote
0answers
30 views

Estimation of transition probability matrix (TPM) for a discrete time, continuous state markov chain from uniformly-spaced samples

I have uniformly spaced samples from a three-component (i.e. three nodes) Markov chain: $s^{(0)}=\begin{bmatrix}0.99\\ 0.01\\ 0.00\end{bmatrix}$, $s^{(1)}=\begin{bmatrix}0.98\\ 0.01\\ 0.01\end{...
6
votes
1answer
93 views

Optimal scaling of the Random Walk Metroplis-Hastings algorithm and the speed measure of the limiting diffusion

Let $d\in\mathbb N$ with $d>1$ $\ell>0$ $\sigma_d^2:=\frac{\ell^2}{d-1}$ $f\in C^2(\mathbb R)$ be positive with $$\int f(x)\:{\rm d}x=1$$ and $g:=\ln f$ $Q_d$ be a Markov kernel on $(\mathbb R^...
0
votes
0answers
5 views

Residence times of the telegraph process ?

The telegraph process is a two state stochastic process defined by the master equation $$ \dot{\pi}_0(t) = \tau^{-1} \pi_1(t) - \sigma^{-1} \pi_0(t) $$ $$ \dot{\pi}_1(t) = \sigma^{-1} \pi_0(t) - \...
3
votes
0answers
38 views

Estimating a MS-ARMA(p,q)-GARCH(r,s) parameters via MCMC

I am currently working on a MS-ARMA-GARCH model proposed by Dhiman das on this paper, and trying to fit it on simulated data. So far I understand the model and its construction, but I'm having a hard ...
1
vote
0answers
19 views

How to create the initial ensemble samples for EnKF

As we know, for the ensemble Kalman filter (EnKF), we need to create a set of samples in the beginning and then to run the predict and analysis step. But for now I have a question of how to create the ...
0
votes
0answers
28 views

Formulating a Transition matrix for Markov Process

I am dealing with a medical process which is as follows. There are 10000 Veterans who are enrolled in this study. All 10000 have medical condition called onychocryptosis which is a fancy term for ...
1
vote
0answers
14 views

Convergence criterion for R-learning algorithm

I'm trying to find a policy for a simple game using R-learning algorithm. I have a field with values (agent can move in 4 directions) and the goal is to get from starting point to finish point with ...
1
vote
1answer
19 views

Distribution of Conditional Brownian Motion

Let $\ X(t),t \ge 0$ be a Brownian motion process. That is, $\ X(t)$ is a process with independent increments such that: $$\ X(t) - X(s) \sim N(0,t-s), 0\le s \lt t $$ and $\ X(0)= 0$. ...
1
vote
1answer
118 views

Markov Chain and Removal Effect

Based on this article I'm trying to use within R the Channel Attribution package to leverage on the Markov Chain in order to attribute conversion between several marketing channel. On one point the ...
0
votes
0answers
19 views

How to calculate probability of a sequence of observations in Hidden markov model

How to calculate the probability of the following sentence - I am I am I
2
votes
0answers
27 views

Justification of acceptance probability in simulated annealing

In simulated annealing the acceptance probability for a new state in step $k$ is traditionally defined as $$ P(\text{accept new})= \begin{cases} \exp(-\frac{\Delta}{T_k}), & \text{ if } \Delta \...
1
vote
0answers
13 views

General techniques for coupling a set of random variables with mutual dependence

Disclaimer: the usage of coupling is in the title is not of the usual definition in probability theory. Suppose I have a set of random variables $\{X_1, X_2, \dots, X_n\}$, indexed by time $t$, and ...
0
votes
0answers
11 views

Markov Decision Process - Need step by step help

I need a step by step solution for this problem: Mainly for Q2 and Q3 The answer for q2 appears to be $R(A)=\sum_{i}^{inf}\left(3*\gamma^{2i+1}+0*\gamma^{2i}\right)=3\gamma$ $\sum_{k}^{inf}\left(\...
0
votes
0answers
28 views

State prediction of how long someone sleeps using neural nets

I have over hundred thousands of datapoints on how long individual people sleep. I also have information about how soft their beds are, their income, stress levels etc. At first I want to predicted ...
2
votes
1answer
77 views

What is the difference betwen a time non-homogenous Markov Chain and a non-linear Markov Chain? Example

A time non-homogenous Markov Chain is one in which the transition probabilities are not constant over time. A non-linear Markov Chain is a model that is not linear in parameters and satisfies the ...
0
votes
1answer
23 views

Markov chain inference confusion

I have a Markov chain where: A->B->C->D I am told that P(A,C,D|B) = P(A|B)P(C,D|B) I am unable to prove why this is the case. Why is this as such?
0
votes
0answers
16 views

MDP value iteration implemenation not yielding proper results?

I am attempting to find optimal policies and state value functions for the following MDP using my implementation in Python, however, I am not quite sure where my code is going wrong or what exactly it ...
0
votes
0answers
15 views

Finite irreducible Markov Chain are recurrent

Question: Show that all state in a finite irreducible Markov chain are recurrent. Attempt: First I considered that a finite irreducible Markov chain is transient. since there are only a finite ...
0
votes
0answers
42 views

expected time to enter a state in birth-death process

I have a question regrading the expected time of entering a state in a birth-death process. Specifically I don't quite understand Page 378 Equation 6.3 of the book here. It is about birth-death ...
-1
votes
0answers
16 views

Lumping states of a markov chain

In my problem I have a matrix describing the transition probabilities between the states of a discrete Markov chain. What I would like to do is to use this information to create groups of states ...
0
votes
0answers
15 views

Estimation of infinitesimal generator/transition rate matrix from proportion data

Suppose I have a collection of data $\{\boldsymbol x_t \in \mathbb S^d\}_{t = 1,\dots,T}$ where $\mathbb S^d$ is the $d$-dimensional unit simplex, i.e. the elements of $\boldsymbol x_t$ sum to $1$. ...
2
votes
0answers
23 views

Markov Chain order 1 vs. AR(1) … Difference and Implication for Parameter Estimation

As other posts on this site indicate, the difference between a time-homogeneous Markov Chain of order 1 and an AR(1) model is merely the assumption of i.i.d. errors, an assumption that we make in AR(1)...
5
votes
1answer
78 views

Memoryless Property of a Markov Chain of Order 1. Is AR(1) memoryless or of infinite memory?

A stochastic process constitutes a discrete Markov Chain of order 1 if it has the memoryless property, in the sense that the probability that the chain will be in a particular state i, of a finite set ...
2
votes
1answer
47 views

Test statistic for lopsidedness of transition matrix

I'm trying to figure out how to estimate, given a transition matrix for a stream of distinct things, what the p-value is that the underlying stream is memoryless. In order to simplify the problem, I'm ...
0
votes
0answers
13 views

How to identify irreducible states by looking at a markov transition matrix?

I'm trying to find a simple way to look at a markov transition matrix and determine the subset of the states which form a closed, irreducible set of states. I came up with the following: If a set of ...
1
vote
1answer
29 views

Markov property definition

The definition of the Markov property is typically that the next state depends only on the present state and no past states. However, the mathematical definition I usually see (e.g. https://stats....
-1
votes
0answers
61 views

Markov chain: Show that states 0, . . N are essential and communicating with each other. What is the difference between essential and communicating?

Markov chain: Show that all states 0, 1, . . . , N are essential and communicating with each other. What is the difference between being essential and communicating in this case? By the definition ...
0
votes
1answer
28 views

Decision making in different intervals in MDPs

I want to model a problem as an MDP model where every day is divided into small time slots (for example minutes) and two decisions A and ...
-1
votes
0answers
14 views

Easy way to sample a Bayes posterior distribution of stable distributions?

I have a markov chain $P(x_{i+1}|x_i)=\rho(x_{i+1} ; \alpha,\beta,c, x_i)$, where $\rho$ is the stable distribution with mean $x_i$. I'm interested in fixing $x_1$ and $x_3$, and sampling an $x_2$ ...
3
votes
1answer
105 views

If the Markov assumption is wrong, will a learner still converge to a stable policy?

I'm trying to figure out what guarantees can be made if a learner wrongly assumes a problem obeys the Markov transition property. Assume I have a problem defined by a partially observable Markov ...
0
votes
0answers
20 views

Initial state in multi-state model

I'm new on the topic of survival analysis and currently I have to build a multi-state model. I've read the articles of the msm and mstate R packages in the Journal of Statistical Software (msm, mstate)...
1
vote
1answer
35 views

Can some one explain me what is difference between Markov process and Markov Decision Process

Markov Process : A stochastic process has Markov property if conditional probability distribution of future states of process depends only upon present state and not on the sequence of events that ...
1
vote
0answers
20 views

Bound for the bias of ergodic averages for non-stationary Markov chains

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $(\mathcal F_n)_{n\in\mathbb N_0}$ be a filtration on $(\Omega,\mathcal A)$ $(E,\mathcal E)$ be a measurable space $X$ be a $(E,\...
0
votes
0answers
13 views

Prove that value function is unique fixed point of bellman operator?

Given an MDP $(S,A,P,\gamma,R)$, the bellman operator is as follows: $$V(s)\mapsto R(s)+\gamma\cdot P_{\pi,s} V$$ Where $P_{\pi,s} V= \sum _{s'\in S}P(s'|s,\pi(s))\cdot V(s)$. Apparently, the value ...
0
votes
0answers
20 views

Multiple Imputation of time series data

I have many groups with a different number of members (learners). The members of each group came together in different time intervals whereupon not all members took part in each of their group ...
1
vote
1answer
39 views

Trouble understanding value iteration

I have trouble understanding how the value iteration algorithm for MDP:s work. I'm trying to follow the canonical grid world example (slide 17), but I don't get the correct results. Here's my work: ...
0
votes
0answers
13 views

Alternative to causal graphs for representing spatial structure in a markov process?

This question is about how to formalize a particular structure of MDP's in an AI/Machine learning context. Consider a markov decision process in a reinforcement learning context. Causal graphs can be ...
0
votes
0answers
13 views

Sensitivity analysis of transition probabilities in a Markov chain

Does anyone know of a method of sensitivity analysis for investigating the effect of perturbing transition probabilities $p_{ij}$ from a Markov transition matrix? I have a series of n=400 sequences ...