Questions tagged [markov-process]
A stochastic process with the property that the future is conditionally independent of the past, given the present.
1,024
questions
0
votes
0answers
20 views
Branching process - extinction probability
So I'm reading about a topic called "Branching process" and I don't completely understand some stuff. Here's what's written in the textbook:
Let $(X_n)_{n \geq 0}$ be a stochastic process where :
$...
0
votes
0answers
25 views
Disease modelling: SIS , SIR , SIR Reed-Frost Models explanations of probabilities
I am reading my textbook and I don't seem to understand some stuff. Here is what is written in the textbook:
Consider a population of $N$ people. There are $3$ different classifications of each ...
1
vote
0answers
19 views
Edited — Normalizing Constant Z in Markov Random Fields
I want to implement this paper in MATLAB. This is the formula:
$$ P(L)=\prod_{s \in S} \frac{1}{Z} exp(- \sum_{c \in N^w(s)}V_c(L_s)) $$
However, I confused to compute $Z$ (normalizing constant) as ...
0
votes
0answers
5 views
Proof by induction of a stationary distribution [closed]
How can i prove by induction that :
$pi$(i)=$pi$(0)/2^i
holds for all i.This is the result of the stationary distribution if a transition matrix.
0
votes
0answers
12 views
Reducible Markov chain with one of the pi entires in the pi vector as zero
I have a reducible, finite markov chain with 2 absorbing states, when finding the stationary distribution I got one of the pi's of the pi vector to be zero.
Does this mean that the Markov chain does ...
0
votes
1answer
31 views
Predicting a Markov chain next state using previously predicted states
Suppose we have a Markov chain with two states A and B.
This associated transition matrix is:
\begin{equation}
P_{mc}=
\begin{...
0
votes
0answers
19 views
How do to determine if a Markov chain is closed
I am struggling to understand the definition of a closed Markov Chain. Does the chain have to have smaller subset that cannot communicate with another states?or if all states communicate does that ...
0
votes
1answer
28 views
Number of stationary distributions of a Markov chain
How do i determine the number of stationary distributions that a Markov chain has if it is not irreducible or regular.
The transition matrix is
...
0
votes
2answers
36 views
Value function and reward function where terminal states have reward 0 and 1
Suppose you have a markov process that can generate bounded random walks such that you have two terminal states... The left most state and the right most state. If the sequence ends in the leftmost ...
2
votes
1answer
102 views
Estimating model for transition probabilities of a Markov Chain
Suppose that I have a Markov chain with $S$ states evolving over time. I have $S^2\times T$ values of the transition matrix, where $T$ is the number of time periods. I also have $K$ matrices $X$ of $T\...
0
votes
0answers
20 views
Markov chain with stopping times
I have a Markov chain with transition matrix $P$, with transition probabilities:
$$p_{i,j}=
\begin{cases}
1-d, & \text{if $i=j \gt 0$} \\[2ex]
d , & \text{ if $j=i-1 \gt 0$} \\[2ex]
(1- \...
0
votes
0answers
35 views
Proving that the uniformly distributed stopping time doesnt have the memoryless property
Consider a chain which is not Markov that waits a time $T^{*}$ before leaving the current state, where $T^{*}$ has uniform distribution over the set of times $\{1, 2, 3, 4\}$ . I would like to show ...
0
votes
1answer
28 views
Problems with using Gibbs Sampling for Bayesian DAGs
Assume we want to sample from the variables of Bayesian belief network, which is a Directed Acyclic Graph (DAG), where we observe some of the variables, and do not observe the others. We can usually ...
1
vote
1answer
53 views
Intuitive explanation of why a stochastic process constructed by a uniformly distributed stopping time is not Markov
Consider a Markov chain that waits a
time $T^{∗}$ before leaving the current state, where $T^{∗}$ has uniform distribution over the set of times $\{1, 2, 3, 4\}$
If $(W_k)_{k\geq}$ would be a ...
0
votes
1answer
95 views
Parsimonious model for transition probabilities for an ordinal Markov chain
I have a time series of an ordinal variable that I wish to model as a first-order Markov chain and estimate the matrix of transition probabilities. (I'm assuming the chain meets all the conditions to ...
2
votes
1answer
95 views
How to show that the transition probability is equal to $\overline p_{ij} = \frac{P_{ij}}{\sum_{k\neq i}p_{ik}}$
(No new answers needed) I would like to award @whuber for his good answer with my bounty!
Suppose that $(X_n)_{n≥0}$ is Markov$(λ, P)$ but
that we only observe the process when it moves to a new ...
0
votes
1answer
36 views
Embedded markov chain example
I have an example of in my textbook of an "embeded markov chain", where I don't understand one step.
Suppose that $(X_n)_{n\geq 0}$ is Markov$(\lambda, P)$. $\lambda$ is the initial distribution and ...
2
votes
0answers
92 views
Can we use of the Metropolis-Hastings importance sampling estimator here?
EDIT: Note that proposing according to $\hat Q$ is equivalent to the following scheme: Given the current state $\tilde x=(i,x')\in\tilde E:=I\times E'$ and $x:=\varphi_i(x')$, sample $j\sim\tilde T(\...
0
votes
0answers
11 views
How to Input to Neural Network Based on Discretely Observed Processes
I'm attempting to estimate the posterior mean for a set of parameters governing a Markov process, using a neural network to minimize squared error (and uninformative priors, along the lines of http://...
0
votes
0answers
33 views
Difference between “hitting time” and “first passage time”
In my textbook, the definition of a "hitting time" is as follows:
Let $(X_n)_{n\geq 0}$ be a markov $(\lambda,P)$ [$\lambda$ is the initial distribution and $P$ is the stochastic matrix]. The hitting ...
2
votes
0answers
218 views
Detailed Balance for Hamiltonian Monte Carlo
I am trying to understand the detailed balance proof present in this paper: https://arxiv.org/abs/hep-lat/9208011v2 (page 4).
My question: Why do we consider the volume of a neighborhood of points ...
8
votes
7answers
3k views
Average time ant needs to get out to the woods
An ant has three passages to choose from:
Passage A takes 7 minutes to get ant out of the ant house to the woods.
Passage B takes 8 minutes to get ant back to the starting point where he is.
Passage ...
1
vote
1answer
50 views
Multiplying two markov chains
Let there be two homogenous markov-chains $(X_t)_{t \in \mathbb{N}_0}$ and $(Y_t)_{t \in \mathbb{N}_0}$ with transition matrices $P_X$ and $P_Y$, given as follows:
$P_X
=
\begin{pmatrix}
0 & 1 &...
2
votes
0answers
30 views
Proving/showing that the Markov property holds in discrete time Markov chain example
I am currently studying the textbook Introduction to Modeling and Analysis of Stochastic Systems, Second Edition, by V. G. Kulkarni. In a section on discrete-time Markov chains, the author presents ...
1
vote
1answer
19 views
Rows and columns of the one-step transition probability matrix
I am currently studying the textbook Introduction to Modeling and Analysis of Stochastic Systems, Second Edition, by V. G. Kulkarni. In a section on discrete-time Markov chains, the author introduces ...
0
votes
0answers
30 views
Markov Chain predictive model combined with rare levels in sequential variable
I'm leveraging on two R packages named clickstream and markovchain in order to:
create a Markovian Chain (model) which will be utilized to perform attribution between different marketing channels ...
0
votes
0answers
18 views
Can someone explain how an action reward function is calculated in markov decision process
In his second lecture on Reinforcement Learning, David Silver, writes the expression for reward function(for MDP) as:
Why do we need to calculate the expected value? Because if we are in state s1 and ...
1
vote
0answers
14 views
What is HMM and Viterbi algorithm?
I have to learn what is HMM and Viterbi alogrithm, I search all pages on Google, but I can't understand what is HMM is and what is Viterbi is, if there is very basic and very simple examples/...
1
vote
0answers
19 views
HMM Bayesian vs. non-Bayesian
I aim to use Hidden Markov Model for regime detection in time series. My question might be a little too blurry:
in which cases it is crucial to use Bayesian version of HMM and in which cases it is ...
2
votes
1answer
28 views
Hitting time definition clarification
Defintion of hitting time in my textbook is written as so:
Let $(X_n)_{n\geq 0}$ be a Markov $(\lambda, P)$. The hitting time of a subset $A \subseteq I$ is a stopping time denoted by $H^A$ such that:...
0
votes
1answer
38 views
How can I prove the simple random walk is a Markov process?
I know a simple random walk is defined as $X_t=X_{t-1}+w_t$, but how can I modify this equation is show it is a Markov process?
1
vote
0answers
20 views
How to fit Markov Chain on price time series?
Markov Chains usually deal with discrete states. But price time series is continuous.
Actually we will be considering not the time series itself, but its diffs:
...
2
votes
1answer
101 views
Markov chain as sum of iid random variables
Suppose I have a sequence of iid random variable, $Z_i$ for $i=0,1,2,3...$ such that $$\Bbb P(Z_i=z)=
\begin{cases}
p, &\text{if } z=1 \\
1-p, & \text{if } z=0.
\end{cases}$$
Define $S_k=\...
3
votes
1answer
40 views
Triangular Markov chain question
A triathlon consists of $3$ disciplines: swimming, cycling and running. A triathlete does a training session every day.
However he doesn’t want to pay for professional coaching advice so instead his
...
2
votes
1answer
41 views
Stochastic Process with Stochastic matrix
I have an exercise with an answer which I don't completely understand:
Here's the exercise:
Suppose $p$ is the transition (stochastic) matrix defined by
$$p= \begin{pmatrix}
1-\alpha & \alpha \\...
1
vote
0answers
62 views
Computing Markov Chain Transition Rates Based On Observed Time
Problem:
I'm working with some data that can be represented with states and transitions, so I would like to model it using Markov chains. Based on the data, I know how much time was spent in each ...
1
vote
1answer
17 views
How to model transition probability if action does not lead to a state change (in MDP)?
An MDP (markov decision process) is defined as a set of states $S$, actions space $A$, Transition Probabilities $T$ and Rewards $R$. An action $a$ in a state $s$ usually result in a change of state ...
2
votes
0answers
22 views
Peskun's ordering after raising to a power
Let $P$ and $Q$ be two Markov transition kernel (or matrix) with the same stationary distribution $\pi$, furthermore assume they are all positive （an operator is positive if the spectrum are all non-...
2
votes
0answers
23 views
Measure how good is the discrete approximation of continuous random process?
If continuous random random process approximated with discrete version - how to measure how good is the approximation?
One possible way is to compute and compare the moments. But for some probability ...
2
votes
0answers
22 views
Asymptotic convergence of the Metropolis-Hastings algorithm with a not necessarily positive target density
Consider the Metropolis-Hastings algorithm on a general state space. Let $p$ denote the density of the target distribution $\mu$, $(X_n)_{n\in\mathbb N_0}$ denote the generated Markov chain and $\...
2
votes
1answer
49 views
Proposal Metropolis distribution for complex Bayesian models
In the book Uncertainty Quantification: Theory, Implementation, and Applications, by R.C. Smith, there is a chapter about Bayesian inference. The likelihood is Gaussian, with error variance $\sigma^2$ ...
12
votes
2answers
881 views
Why does thinning work in Bayesian inference?
In Bayesian inference, one needs to determine the posterior distribution of the parameters from the prior distribution and the likelihood of the data. As this computation might not be possible ...
1
vote
0answers
12 views
Holding time in Markov processes
I was trying to solve same past exams regarding stochastic processes and I think that the solutions may be wrong. I am posting here because I would like some clarification if actually are the ...
0
votes
1answer
21 views
What step do they take to solve the limiting distribution
small question. I just can't figure out what step they take to find X1, X2 and X3 (see picture) after I set up the equations.
How do i proceed to go from the 3 diffrent equations to an answer. It ...
1
vote
1answer
29 views
Expectation of dependent Bernoulli variables
Let's say the Bernoulli random variables $X_1,X_2,...,X_n$ follow a Markov process of order 1. Does this imply that for $t=1,...,n$
\begin{equation}
\mathbb{E}(X_t)=P(X_t=1\mid X_{t-1}=x_{t-1})
\end{...
1
vote
1answer
44 views
Unbiased Metropolis-Hastings estimator of the form $\frac{\sum_{i=1}^nW_if(Y_i)}{\sum_{i=1}^nW_i}$. How do we need to choose $W_i$?
Let $(X_n)_{n\in\mathbb N_0}$ be the Markov chain generated by the Metorpolis-Hastings algorithm with proposal kernel $Q$ and target distribution $\mu$ and $(Y_n)_{n\in\mathbb N}$ denote the ...
1
vote
0answers
17 views
Conditioning modified negative binomial likelihood on number of indices
I am simulating a negative binomially distributed branching process (Galton-Watson process) with parameters $R$ (i.e. $\mu$) and $k$ initiated with a single event (terming 'index'). Each branching ...
0
votes
0answers
21 views
Can we use a Dirac kernel in the proposal of the Metropolis-Hastings algorithm?
I'm running the Metropolis-Hastings algorithm on a product space $(I\times E,2^I\otimes\mathcal E,\zeta\otimes\lambda)$, where $I$ is a finite nonempty set and $\zeta$ denotes the counting measure.
...
0
votes
0answers
17 views
Prediction Accuracy of Channel Attribution Models
I have implemented the following methods for channel attribution :
First Touch
Last Touch
1st Order Markov Chain
Shapley Value
Bagged Logistic Regression
I want to compare them on the predictive ...
3
votes
1answer
36 views
Determine if the following Markov chain is positive recurrent, null recurrent or transcient
We consider the Markov Chain with transition probabilities
$$
p(i,0)=\frac{1}{i^2 +2},\qquad p(i,i+1)= \frac{i^2 +1}{i^2 +2}.
$$
Determine if this Markov chain is positive recurrent, null recurrent ...
