Questions tagged [markov-process]
A stochastic process with the property that the future is conditionally independent of the past, given the present.
960
questions
0
votes
0answers
8 views
Minimization of the asymptotic variance of a Metropolis-Hastings waste-recycling estimator
Let $\alpha$ denote the acceptance function of the Markov chain $(X_n)_{n\in\mathbb N_0}$ generated by the Metropolis-Hastings algorithm with proposal kernel $Q$ and target distribution $\mu$$^1$ and $...
0
votes
0answers
5 views
Is the stationary distribution of the augmented Metropolis-Hastings kernel even reversible in the symmetric proposal case?
Let $\alpha$ denote the acceptance function of the Markov chain generated by the Metropolis-Hastings algorithm with proposal kernel $Q$ and target distribution $\mu$$^1$ and $$\kappa_{\text{aug}}((x,y)...
3
votes
1answer
52 views
Central limit theorem for a Metropolis-Hastings estimator
Let $\alpha$ denote the acceptance function of the Markov chain $(X_n)_{n\in\mathbb N_0}$ generated by the Metropolis-Hastings algorithm with proposal kernel $Q$ and target distribution $\mu$$^1$ and $...
0
votes
0answers
11 views
Does this problem satisfy markov properties to be modeled as HMM?
I want to model a chemical reaction network which is defined by a stoichiometric matrix $\nu^{s\times m} $ where $s$ is the number of participating species and $m$ the number of chemical reactions. If ...
0
votes
0answers
17 views
the convergence speed for a Markov chain
For a metropolis hastings algorithm, suppose that the stationary distribution is defined as the Gibbs Boltzmann distribution $\pi_T(x)= \frac{1}{Z_T}e^{-\frac{V(x)}{T} }$ where $Z_T = \sum_{y\in V} e^{...
0
votes
1answer
56 views
Could someone help me to understand the Metropolis-Hastings algorithm for discrete Markov Chains?
Metropolis-Hastings Algorithm
Assume the Markov chain is in some state $X_{n} = i$. Let $\textbf{H}$ be the transition matrix for any irreducible Markov chain on the state space. We generate $X_{n+1}$...
0
votes
1answer
32 views
What is the definition of a aperiodic Markov chain?
I understand the definition of a state being aperiodic or periodic with period d. But what does it mean for a chain to be aperiodic / periodic with period d?
Thanks.
0
votes
0answers
8 views
Quantiles of the Q values of an MDP
Cross-posted from Math StackExchange:
Consider an MDP with $n$ states, $k$ actions, and discount factor $\gamma \in [0,1)$. We are uncertain of its reward function $R \in \mathbb{R}^{n \times k}$ and ...
0
votes
0answers
11 views
Markov Chain Attribution Model: Calendar periods VS Sliding Window?
I'm trying to utilize Markov Chains in order to analyze and attribute online conversions of b2b users browsing on a company web. The key question mark I'm facing is on which period length to apply the ...
2
votes
1answer
106 views
Point process model diagnostic: Nearest-Neighbor Distance Distribution or Pair Correlation Function?
I have a point pattern which is clearly inhomogeneous. Furthermore, the inhomogeneity has two components: a large scale effect and a local scale effect. I have constructed a Markov point process model ...
0
votes
1answer
24 views
Proving a Markov Chain (Random Walk) is Time-Homogeneous
Let $Y_0, Y_1,Y_2,... $ be a sequence of independent and identically distributed random variables. Then we define $X_n = \displaystyle\sum_{j=0}^{n} Y_j $ , $n=0,1,2,...$.
This is a Random Walk ...
0
votes
0answers
10 views
HMM - Deal with Baum-Welch emission never observed
If I train a HMM with a given sequence of observations among n possible emissions, how do I deal with an emission that is never observed? For example, if in a 100 long observation sequence the ...
0
votes
0answers
20 views
Why is this Markov Equivalence true?
I have the following markov relation $X_1 \leftarrow X \rightarrow X_2$, which leads to $X_1 \rightarrow X \rightarrow X_2$, but how does this lead to
$X \rightarrow (X_1, X_2) \rightarrow X_2$
Can ...
0
votes
0answers
11 views
Geometric Brownian Motion with two-state diffusion/volatility
let's assume a GBM process S(t) with dynamics:
dS(t) = a S(t) dt + b S(t) dB(t)
where B(t) is a Brownian motion, a and b are constants, and S(0)>0.
For any time s>t, we have that
E_t[S(s)^k] = S(t)...
0
votes
0answers
12 views
Steady state distributions and stationary proabilities
Whats the difference between steady state distributions, long run probabilities and stationary distributions?
0
votes
0answers
39 views
Markov chain: expected number of visits to a state
Let $X_0$, $X_1$, $X_2$, ... be a Markov chain with state space 0, 1, 2, 3 and transition probabilities
$$
\begin{matrix}
1/2 & 1/2 & 0 & 0\\
1/3 & 1/3 & 1/3 & 0\\
0 &...
1
vote
0answers
15 views
Number of observations required to approximate discrete random variable
I am trying to understand if, given a discrete random variable, there exists a formalised approach for determining the number of observations required to "well approximate" said variables true ...
0
votes
1answer
16 views
probability of getting to state a before state b starting from state a
If $X_n$ is a Markov Chain. P(0,0) = 0.5, P(0,1) = 0.5. For all states x > 0, P(x, x) = 0.5, P(x, x+1) = P(x, x-1) = 0.25. My goal is to find $P_0$ ($T_0$ < $T_5$), which is the probability of ...
0
votes
1answer
47 views
Machine Learning alternative for hashing [closed]
Is there a Machine Learning technique that can used to detect the slightest change in data? I know this can be done using a hash but I was just wondering if there is any machine learning technique out ...
0
votes
1answer
57 views
Markov Chain: transitioning to multiple states at the same time
I'm trying to calculate Customer Lifetime Value using Markov Chains. I'm following the paper by Pfeirer and Carraway.
The paper evaluates CLV over a finite time horizon ...
0
votes
1answer
49 views
Energy function of Restricted Boltzmann Machine (RBM)
The energy function for RBM (Restricted Boltzmann Machine) is defined as
$$
E(v,h) = -\sum_{i,j} w_{ij} \, v_i \, h_j -\sum_i a_i \, v_i - \sum_i b_i \, h_i
$$
with the joint distribution
$$
\tag{1}
p(...
2
votes
0answers
19 views
Model or State Uncertainty in Queueing Model due to uncertain arrival rate
$\textbf{Introduction}$
I am currently modelling a scenario where two queues need to be served by a single server in a non preemptive discipline. I am quite sorted on generating the optimal policy ...
2
votes
1answer
62 views
How can we apply the rule of stationary distribution to the continuous case of Markov chain?
If the Markov chain converged then $$\pi = Q* \pi$$where $ \pi$ is the posterior distribution and $Q$ is the transition distribution(it's a matrix in the discrete case).
I tried to apply that on the ...
2
votes
1answer
44 views
Hidden-Markov Model for Markov-Chain with Sequential Bernoulli State Sampling
Consider a finite discrete-time Markov chain whose state is sampled at the times determined by the outcome of a Bernoulli process. That is, in each time period I flip a biased coin. If it comes up as "...
3
votes
1answer
42 views
Can we calculate the theoretical stationary distribution from a continuous Markov chain?
I have the transition distribution $p(X_{t+1}|X_t=x_t) = \text{N}(\phi x_t,1)$ where $−1<\phi<1$.
Can we calculate the stationary distribution and its mean and variance? I know I can do that ...
0
votes
0answers
85 views
Variance reduction of an estimator arising from the marginal destribution of a Metropolis-Hastings chain
Let
$(E,\mathcal E,\lambda)$ and $(E',\mathcal E',\lambda')$ be measure spaces
$f\in L^2(\lambda)$
$I$ be a finite nonempty set
$\varphi_i:E'\to E$ be bijective $(\mathcal E',\mathcal E)$-measurable ...
1
vote
1answer
48 views
Is the transition kernel of a Metropolis-Hastings chain of the form $P(x,A)=\varrho(x)\tilde P(x,A)+(1-\varrho(x))1_A(x)$?
After equation (1) at page 3 of this paper it is claimed that the transition kernel of a Markov chain generated by the Metropolis-Hastings algorithm is of the form $$P(x,A)=\varrho(x)\tilde P(x,A)+(1-\...
1
vote
1answer
24 views
Birth Death process
An office has two employees that process incoming orders. these two are always busy and they process the orders at the rate of 100/day for each person. However they are smokers. On an average they ...
0
votes
2answers
79 views
Suggest a model for this dataset
I have a time series data set (the Old Faithful geyser data available here: http://www.gatsby.ucl.ac.uk/teaching/courses/ml1-2012/geyser.txt). Plotting the eruption duration on the x axis and the ...
1
vote
2answers
53 views
Could a non-probabilistic state be a random variable?
This post gives a definition "of A stochastic process in discrete time"
A stochastic process in discrete time n ∈ $N$ = {0, 1, 2, . . .} is a sequence of random variables (rvs) $X_0, X_1, X_2$, . . ...
-1
votes
1answer
40 views
What is the n of Markov chain exactly equal to?
Section 7.2 of the book "transition probability graph" coming from the book "Introduction to Probability, 2nd Edition by Dimitri P. Bertsekas and John N. Tsitsiklis" gives some explanation of the ...
0
votes
1answer
28 views
Is there a name for this type of transition probability diagram which seems not to be a transition probability graph?
This is a "transition probability graph" coming from the book "Introduction to Probability, 2nd Edition by Dimitri P. Bertsekas and John N. Tsitsiklis".
That book also gives this figure,
which seems ...
0
votes
0answers
15 views
How many Markov chains are there for 2 states in 1, 2 and 3 steps?
wiki uses this example to illustrate Markov chains.
The probabilities of weather conditions (modeled as either rainy or
sunny), given the weather on the preceding day, can be represented by
a ...
0
votes
1answer
94 views
What is the minimum of n to construct a Markov chain?
Wiki gives this definition of a discrete-time Markov chain
a sequence of random variables $X_1$, $X_2$, $X_3$, ... with the Markov property, namely that the probability of moving to the next state ...
0
votes
1answer
19 views
Given a transition matrix P for weather conditions (modeled as either rainy or sunny), is $P^n$ the n-Step Transition Probabilities for day n+1?
wiki uses this example to illustrate Markov chains.
The probabilities of weather conditions (modeled as either rainy or
sunny), given the weather on the preceding day, can be represented by
a ...
0
votes
1answer
24 views
When to exponentiate: the mean of the chain or at every step in the chain?
I am interested in when it is best to exponentiate a difference in log-odds
Here is a sample problem in the stan language, three groups of forty binary observations, group 1 with hit probability = 0....
0
votes
0answers
59 views
Make sense of plotting a transition matrix
I'm studying statistics and I'm trying to understand markov chain topic. I'm using the package "markovchain" in R to obtain the stationary distribution.
From this transition matrix $M$:
...
0
votes
0answers
11 views
Approximate Value / Policy Iteration (Reinforcement Learning)
I am reading Markov Decision Processes in AI : about Approximate Dynamic Programming.
Would you like to explain the rationale for introducing the API algorithm, how it compares to AVI ?
How would ...
2
votes
0answers
27 views
Is there a reason why we should run the Metorpolis-Hastings algorithm with a target density approximating the density we're actually after?
Let $(E,\mathcal E,\lambda)$ be a measure space, $p:E\to[0,\infty)$ be $\mathcal E$-measurable with $$c:=\int p\:{\rm d}\lambda$$ and $$\mu:=\underbrace{\frac1cp}_{=:\:\tilde p}\lambda$$ denote the ...
1
vote
0answers
13 views
Can the interarrival times of a continuous time markov chain be distributed with 2 parameter (scale,location) exponential distributions?
I'm trying to model data with a time-homogenous CTMC with a number of states with corresponding constant transition rates $\lambda_{i}$ when I notice that much of the transition times from one state ...
0
votes
0answers
12 views
Markov chain transition Matrix for Ion channels in Python
I was working to code for a markov chain transition matrix for potassium channels. Potassium channels conists of 4 gates 1,2,3,4. the potassium ions antransition from 1->2 or 1->1, 2->1,2->2,2->3 etc. ...
0
votes
1answer
21 views
HMM rolling estimation different from batch estimation
I'm using the GuassianHMM from the python package hmmlearn
and after fitting the hmm to the data the predictions that are done in one batch
...
1
vote
0answers
50 views
Sampling with fixed probability from two different distributions. How is the sample distributed?
Let
$(\Omega,\mathcal A,\operatorname P)$ be a probability space
$\mu$ be a probability measure on $(\mathbb R,\mathcal B(\mathbb R))$
$X$ be real-valued random variable on $(\Omega,\mathcal A,\...
1
vote
0answers
31 views
How to calculate a 𝑛-step transitions of a Discrete-time Markov Chain for Figure 17.1 (b) in book “Machine Learning - A Probabilistic Perspective”
chapter 17 of the book "Machine Learning - A Probabilistic Perspective" gives this figure
which is the probability of getting from i to j in exactly n steps.
Obviously A(1) = A.
In the case of ...
0
votes
0answers
37 views
What is the point of doing simulation on Markov Chain?
I am studying Markov Chain and I am currently reading about simulation on Markov Chain but I can't see the point of simulation on Markov Chain. What does simulation mean in Markov Chain and what can ...
1
vote
1answer
27 views
can the prior probability change with time?
I was reading Markov Models for sequence modeling and stuck with my understanding(hypothesis).
In Baye's theorem, can the prior probabilities change with time? If the prior probability at t=3 is 0.05,...
0
votes
0answers
59 views
Measure the distance between two probability transition matrices
I have a probability transition matrix $P$ that contains some values very close to zero. I want to sparsify this matrix by taking the k largest values for each row and setting the others to zero. For ...
0
votes
0answers
12 views
Markov Chain Monte Carlo for Decryption Purposes
I've been reading the "Markov Chain Monte Carlov Revolution" paper by Diaconis and was intrigued by his application to decryption. I've done some further reading and am now trying to work through ...
1
vote
1answer
17 views
What are the necessary qualifications or assumptions to say that a graph structure is a Markov Chain?
I have a graph structure and want to say it is a Markov Chain. But I am wondering what necessary assumptions or properties that my graph structure need to meet to be called a Markov Chain?
0
votes
0answers
6 views
What are the relationships among Markov Property, Stationarity, and Time Invariance
I am wondering if there is or are any relationship among those. I have understood Markov Property by reading Wikipedia, but it is still confusing to figure out if there is any relationship among those ...
