Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

Filter by
Sorted by
Tagged with
0
votes
0answers
20 views

Branching process - extinction probability

So I'm reading about a topic called "Branching process" and I don't completely understand some stuff. Here's what's written in the textbook: Let $(X_n)_{n \geq 0}$ be a stochastic process where : $...
0
votes
0answers
25 views

Disease modelling: SIS , SIR , SIR Reed-Frost Models explanations of probabilities

I am reading my textbook and I don't seem to understand some stuff. Here is what is written in the textbook: Consider a population of $N$ people. There are $3$ different classifications of each ...
1
vote
0answers
19 views

Edited — Normalizing Constant Z in Markov Random Fields

I want to implement this paper in MATLAB. This is the formula: $$ P(L)=\prod_{s \in S} \frac{1}{Z} exp(- \sum_{c \in N^w(s)}V_c(L_s)) $$ However, I confused to compute $Z$ (normalizing constant) as ...
0
votes
0answers
5 views

Proof by induction of a stationary distribution [closed]

How can i prove by induction that : $pi$(i)=$pi$(0)/2^i holds for all i.This is the result of the stationary distribution if a transition matrix.
0
votes
0answers
12 views

Reducible Markov chain with one of the pi entires in the pi vector as zero

I have a reducible, finite markov chain with 2 absorbing states, when finding the stationary distribution I got one of the pi's of the pi vector to be zero. Does this mean that the Markov chain does ...
0
votes
1answer
31 views

Predicting a Markov chain next state using previously predicted states

Suppose we have a Markov chain with two states A and B. This associated transition matrix is: \begin{equation} P_{mc}= \begin{...
0
votes
0answers
19 views

How do to determine if a Markov chain is closed

I am struggling to understand the definition of a closed Markov Chain. Does the chain have to have smaller subset that cannot communicate with another states?or if all states communicate does that ...
0
votes
1answer
28 views

Number of stationary distributions of a Markov chain

How do i determine the number of stationary distributions that a Markov chain has if it is not irreducible or regular. The transition matrix is ...
0
votes
2answers
36 views

Value function and reward function where terminal states have reward 0 and 1

Suppose you have a markov process that can generate bounded random walks such that you have two terminal states... The left most state and the right most state. If the sequence ends in the leftmost ...
2
votes
1answer
102 views

Estimating model for transition probabilities of a Markov Chain

Suppose that I have a Markov chain with $S$ states evolving over time. I have $S^2\times T$ values of the transition matrix, where $T$ is the number of time periods. I also have $K$ matrices $X$ of $T\...
0
votes
0answers
20 views

Markov chain with stopping times

I have a Markov chain with transition matrix $P$, with transition probabilities: $$p_{i,j}= \begin{cases} 1-d, & \text{if $i=j \gt 0$} \\[2ex] d , & \text{ if $j=i-1 \gt 0$} \\[2ex] (1- \...
0
votes
0answers
35 views

Proving that the uniformly distributed stopping time doesnt have the memoryless property

Consider a chain which is not Markov that waits a time $T^{*}$ before leaving the current state, where $T^{*}$ has uniform distribution over the set of times $\{1, 2, 3, 4\}$ . I would like to show ...
0
votes
1answer
28 views

Problems with using Gibbs Sampling for Bayesian DAGs

Assume we want to sample from the variables of Bayesian belief network, which is a Directed Acyclic Graph (DAG), where we observe some of the variables, and do not observe the others. We can usually ...
1
vote
1answer
53 views

Intuitive explanation of why a stochastic process constructed by a uniformly distributed stopping time is not Markov

Consider a Markov chain that waits a time $T^{∗}$ before leaving the current state, where $T^{∗}$ has uniform distribution over the set of times $\{1, 2, 3, 4\}$ If $(W_k)_{k\geq}$ would be a ...
0
votes
1answer
95 views

Parsimonious model for transition probabilities for an ordinal Markov chain

I have a time series of an ordinal variable that I wish to model as a first-order Markov chain and estimate the matrix of transition probabilities. (I'm assuming the chain meets all the conditions to ...
2
votes
1answer
95 views

How to show that the transition probability is equal to $\overline p_{ij} = \frac{P_{ij}}{\sum_{k\neq i}p_{ik}}$

(No new answers needed) I would like to award @whuber for his good answer with my bounty! Suppose that $(X_n)_{n≥0}$ is Markov$(λ, P)$ but that we only observe the process when it moves to a new ...
0
votes
1answer
36 views

Embedded markov chain example

I have an example of in my textbook of an "embeded markov chain", where I don't understand one step. Suppose that $(X_n)_{n\geq 0}$ is Markov$(\lambda, P)$. $\lambda$ is the initial distribution and ...
2
votes
0answers
92 views

Can we use of the Metropolis-Hastings importance sampling estimator here?

EDIT: Note that proposing according to $\hat Q$ is equivalent to the following scheme: Given the current state $\tilde x=(i,x')\in\tilde E:=I\times E'$ and $x:=\varphi_i(x')$, sample $j\sim\tilde T(\...
0
votes
0answers
11 views

How to Input to Neural Network Based on Discretely Observed Processes

I'm attempting to estimate the posterior mean for a set of parameters governing a Markov process, using a neural network to minimize squared error (and uninformative priors, along the lines of http://...
0
votes
0answers
33 views

Difference between “hitting time” and “first passage time”

In my textbook, the definition of a "hitting time" is as follows: Let $(X_n)_{n\geq 0}$ be a markov $(\lambda,P)$ [$\lambda$ is the initial distribution and $P$ is the stochastic matrix]. The hitting ...
2
votes
0answers
218 views

Detailed Balance for Hamiltonian Monte Carlo

I am trying to understand the detailed balance proof present in this paper: https://arxiv.org/abs/hep-lat/9208011v2 (page 4). My question: Why do we consider the volume of a neighborhood of points ...
8
votes
7answers
3k views

Average time ant needs to get out to the woods

An ant has three passages to choose from: Passage A takes 7 minutes to get ant out of the ant house to the woods. Passage B takes 8 minutes to get ant back to the starting point where he is. Passage ...
1
vote
1answer
50 views

Multiplying two markov chains

Let there be two homogenous markov-chains $(X_t)_{t \in \mathbb{N}_0}$ and $(Y_t)_{t \in \mathbb{N}_0}$ with transition matrices $P_X$ and $P_Y$, given as follows: $P_X = \begin{pmatrix} 0 & 1 &...
2
votes
0answers
30 views

Proving/showing that the Markov property holds in discrete time Markov chain example

I am currently studying the textbook Introduction to Modeling and Analysis of Stochastic Systems, Second Edition, by V. G. Kulkarni. In a section on discrete-time Markov chains, the author presents ...
1
vote
1answer
19 views

Rows and columns of the one-step transition probability matrix

I am currently studying the textbook Introduction to Modeling and Analysis of Stochastic Systems, Second Edition, by V. G. Kulkarni. In a section on discrete-time Markov chains, the author introduces ...
0
votes
0answers
30 views

Markov Chain predictive model combined with rare levels in sequential variable

I'm leveraging on two R packages named clickstream and markovchain in order to: create a Markovian Chain (model) which will be utilized to perform attribution between different marketing channels ...
0
votes
0answers
18 views

Can someone explain how an action reward function is calculated in markov decision process

In his second lecture on Reinforcement Learning, David Silver, writes the expression for reward function(for MDP) as: Why do we need to calculate the expected value? Because if we are in state s1 and ...
1
vote
0answers
14 views

What is HMM and Viterbi algorithm?

I have to learn what is HMM and Viterbi alogrithm, I search all pages on Google, but I can't understand what is HMM is and what is Viterbi is, if there is very basic and very simple examples/...
1
vote
0answers
19 views

HMM Bayesian vs. non-Bayesian

I aim to use Hidden Markov Model for regime detection in time series. My question might be a little too blurry: in which cases it is crucial to use Bayesian version of HMM and in which cases it is ...
2
votes
1answer
28 views

Hitting time definition clarification

Defintion of hitting time in my textbook is written as so: Let $(X_n)_{n\geq 0}$ be a Markov $(\lambda, P)$. The hitting time of a subset $A \subseteq I$ is a stopping time denoted by $H^A$ such that:...
0
votes
1answer
38 views

How can I prove the simple random walk is a Markov process?

I know a simple random walk is defined as $X_t=X_{t-1}+w_t$, but how can I modify this equation is show it is a Markov process?
1
vote
0answers
20 views

How to fit Markov Chain on price time series?

Markov Chains usually deal with discrete states. But price time series is continuous. Actually we will be considering not the time series itself, but its diffs: ...
2
votes
1answer
101 views

Markov chain as sum of iid random variables

Suppose I have a sequence of iid random variable, $Z_i$ for $i=0,1,2,3...$ such that $$\Bbb P(Z_i=z)= \begin{cases} p, &\text{if } z=1 \\ 1-p, & \text{if } z=0. \end{cases}$$ Define $S_k=\...
3
votes
1answer
40 views

Triangular Markov chain question

A triathlon consists of $3$ disciplines: swimming, cycling and running. A triathlete does a training session every day. However he doesn’t want to pay for professional coaching advice so instead his ...
2
votes
1answer
41 views

Stochastic Process with Stochastic matrix

I have an exercise with an answer which I don't completely understand: Here's the exercise: Suppose $p$ is the transition (stochastic) matrix defined by $$p= \begin{pmatrix} 1-\alpha & \alpha \\...
1
vote
0answers
62 views

Computing Markov Chain Transition Rates Based On Observed Time

Problem: I'm working with some data that can be represented with states and transitions, so I would like to model it using Markov chains. Based on the data, I know how much time was spent in each ...
1
vote
1answer
17 views

How to model transition probability if action does not lead to a state change (in MDP)?

An MDP (markov decision process) is defined as a set of states $S$, actions space $A$, Transition Probabilities $T$ and Rewards $R$. An action $a$ in a state $s$ usually result in a change of state ...
2
votes
0answers
22 views

Peskun's ordering after raising to a power

Let $P$ and $Q$ be two Markov transition kernel (or matrix) with the same stationary distribution $\pi$, furthermore assume they are all positive (an operator is positive if the spectrum are all non-...
2
votes
0answers
23 views

Measure how good is the discrete approximation of continuous random process?

If continuous random random process approximated with discrete version - how to measure how good is the approximation? One possible way is to compute and compare the moments. But for some probability ...
2
votes
0answers
22 views

Asymptotic convergence of the Metropolis-Hastings algorithm with a not necessarily positive target density

Consider the Metropolis-Hastings algorithm on a general state space. Let $p$ denote the density of the target distribution $\mu$, $(X_n)_{n\in\mathbb N_0}$ denote the generated Markov chain and $\...
2
votes
1answer
49 views

Proposal Metropolis distribution for complex Bayesian models

In the book Uncertainty Quantification: Theory, Implementation, and Applications, by R.C. Smith, there is a chapter about Bayesian inference. The likelihood is Gaussian, with error variance $\sigma^2$ ...
12
votes
2answers
881 views

Why does thinning work in Bayesian inference?

In Bayesian inference, one needs to determine the posterior distribution of the parameters from the prior distribution and the likelihood of the data. As this computation might not be possible ...
1
vote
0answers
12 views

Holding time in Markov processes

I was trying to solve same past exams regarding stochastic processes and I think that the solutions may be wrong. I am posting here because I would like some clarification if actually are the ...
0
votes
1answer
21 views

What step do they take to solve the limiting distribution

small question. I just can't figure out what step they take to find X1, X2 and X3 (see picture) after I set up the equations. How do i proceed to go from the 3 diffrent equations to an answer. It ...
1
vote
1answer
29 views

Expectation of dependent Bernoulli variables

Let's say the Bernoulli random variables $X_1,X_2,...,X_n$ follow a Markov process of order 1. Does this imply that for $t=1,...,n$ \begin{equation} \mathbb{E}(X_t)=P(X_t=1\mid X_{t-1}=x_{t-1}) \end{...
1
vote
1answer
44 views

Unbiased Metropolis-Hastings estimator of the form $\frac{\sum_{i=1}^nW_if(Y_i)}{\sum_{i=1}^nW_i}$. How do we need to choose $W_i$?

Let $(X_n)_{n\in\mathbb N_0}$ be the Markov chain generated by the Metorpolis-Hastings algorithm with proposal kernel $Q$ and target distribution $\mu$ and $(Y_n)_{n\in\mathbb N}$ denote the ...
1
vote
0answers
17 views

Conditioning modified negative binomial likelihood on number of indices

I am simulating a negative binomially distributed branching process (Galton-Watson process) with parameters $R$ (i.e. $\mu$) and $k$ initiated with a single event (terming 'index'). Each branching ...
0
votes
0answers
21 views

Can we use a Dirac kernel in the proposal of the Metropolis-Hastings algorithm?

I'm running the Metropolis-Hastings algorithm on a product space $(I\times E,2^I\otimes\mathcal E,\zeta\otimes\lambda)$, where $I$ is a finite nonempty set and $\zeta$ denotes the counting measure. ...
0
votes
0answers
17 views

Prediction Accuracy of Channel Attribution Models

I have implemented the following methods for channel attribution : First Touch Last Touch 1st Order Markov Chain Shapley Value Bagged Logistic Regression I want to compare them on the predictive ...
3
votes
1answer
36 views

Determine if the following Markov chain is positive recurrent, null recurrent or transcient

We consider the Markov Chain with transition probabilities $$ p(i,0)=\frac{1}{i^2 +2},\qquad p(i,i+1)= \frac{i^2 +1}{i^2 +2}. $$ Determine if this Markov chain is positive recurrent, null recurrent ...

1 2 3 4 5 21