The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

Filter by
Sorted by
Tagged with
0
votes
0answers
8 views

Minimization of the asymptotic variance of a Metropolis-Hastings waste-recycling estimator

Let $\alpha$ denote the acceptance function of the Markov chain $(X_n)_{n\in\mathbb N_0}$ generated by the Metropolis-Hastings algorithm with proposal kernel $Q$ and target distribution $\mu$$^1$ and $...
0
votes
0answers
5 views

Is the stationary distribution of the augmented Metropolis-Hastings kernel even reversible in the symmetric proposal case?

Let $\alpha$ denote the acceptance function of the Markov chain generated by the Metropolis-Hastings algorithm with proposal kernel $Q$ and target distribution $\mu$$^1$ and $$\kappa_{\text{aug}}((x,y)...
3
votes
1answer
52 views

Central limit theorem for a Metropolis-Hastings estimator

Let $\alpha$ denote the acceptance function of the Markov chain $(X_n)_{n\in\mathbb N_0}$ generated by the Metropolis-Hastings algorithm with proposal kernel $Q$ and target distribution $\mu$$^1$ and $...
0
votes
0answers
11 views

Does this problem satisfy markov properties to be modeled as HMM?

I want to model a chemical reaction network which is defined by a stoichiometric matrix $\nu^{s\times m} $ where $s$ is the number of participating species and $m$ the number of chemical reactions. If ...
0
votes
0answers
17 views

the convergence speed for a Markov chain

For a metropolis hastings algorithm, suppose that the stationary distribution is defined as the Gibbs Boltzmann distribution $\pi_T(x)= \frac{1}{Z_T}e^{-\frac{V(x)}{T} }$ where $Z_T = \sum_{y\in V} e^{...
0
votes
1answer
56 views

Could someone help me to understand the Metropolis-Hastings algorithm for discrete Markov Chains?

Metropolis-Hastings Algorithm Assume the Markov chain is in some state $X_{n} = i$. Let $\textbf{H}$ be the transition matrix for any irreducible Markov chain on the state space. We generate $X_{n+1}$...
0
votes
1answer
32 views

What is the definition of a aperiodic Markov chain?

I understand the definition of a state being aperiodic or periodic with period d. But what does it mean for a chain to be aperiodic / periodic with period d? Thanks.
0
votes
0answers
8 views

Quantiles of the Q values of an MDP

Cross-posted from Math StackExchange: Consider an MDP with $n$ states, $k$ actions, and discount factor $\gamma \in [0,1)$. We are uncertain of its reward function $R \in \mathbb{R}^{n \times k}$ and ...
0
votes
0answers
11 views

Markov Chain Attribution Model: Calendar periods VS Sliding Window?

I'm trying to utilize Markov Chains in order to analyze and attribute online conversions of b2b users browsing on a company web. The key question mark I'm facing is on which period length to apply the ...
2
votes
1answer
106 views

Point process model diagnostic: Nearest-Neighbor Distance Distribution or Pair Correlation Function?

I have a point pattern which is clearly inhomogeneous. Furthermore, the inhomogeneity has two components: a large scale effect and a local scale effect. I have constructed a Markov point process model ...
0
votes
1answer
24 views

Proving a Markov Chain (Random Walk) is Time-Homogeneous

Let $Y_0, Y_1,Y_2,... $ be a sequence of independent and identically distributed random variables. Then we define $X_n = \displaystyle\sum_{j=0}^{n} Y_j $ , $n=0,1,2,...$. This is a Random Walk ...
0
votes
0answers
10 views

HMM - Deal with Baum-Welch emission never observed

If I train a HMM with a given sequence of observations among n possible emissions, how do I deal with an emission that is never observed? For example, if in a 100 long observation sequence the ...
0
votes
0answers
20 views

Why is this Markov Equivalence true?

I have the following markov relation $X_1 \leftarrow X \rightarrow X_2$, which leads to $X_1 \rightarrow X \rightarrow X_2$, but how does this lead to $X \rightarrow (X_1, X_2) \rightarrow X_2$ Can ...
0
votes
0answers
11 views

Geometric Brownian Motion with two-state diffusion/volatility

let's assume a GBM process S(t) with dynamics: dS(t) = a S(t) dt + b S(t) dB(t) where B(t) is a Brownian motion, a and b are constants, and S(0)>0. For any time s>t, we have that E_t[S(s)^k] = S(t)...
0
votes
0answers
12 views

Steady state distributions and stationary proabilities

Whats the difference between steady state distributions, long run probabilities and stationary distributions?
0
votes
0answers
39 views

Markov chain: expected number of visits to a state

Let $X_0$, $X_1$, $X_2$, ... be a Markov chain with state space 0, 1, 2, 3 and transition probabilities $$ \begin{matrix} 1/2 & 1/2 & 0 & 0\\ 1/3 & 1/3 & 1/3 & 0\\ 0 &...
1
vote
0answers
15 views

Number of observations required to approximate discrete random variable

I am trying to understand if, given a discrete random variable, there exists a formalised approach for determining the number of observations required to "well approximate" said variables true ...
0
votes
1answer
16 views

probability of getting to state a before state b starting from state a

If $X_n$ is a Markov Chain. P(0,0) = 0.5, P(0,1) = 0.5. For all states x > 0, P(x, x) = 0.5, P(x, x+1) = P(x, x-1) = 0.25. My goal is to find $P_0$ ($T_0$ < $T_5$), which is the probability of ...
0
votes
1answer
47 views

Machine Learning alternative for hashing [closed]

Is there a Machine Learning technique that can used to detect the slightest change in data? I know this can be done using a hash but I was just wondering if there is any machine learning technique out ...
0
votes
1answer
57 views

Markov Chain: transitioning to multiple states at the same time

I'm trying to calculate Customer Lifetime Value using Markov Chains. I'm following the paper by Pfeirer and Carraway. The paper evaluates CLV over a finite time horizon ...
0
votes
1answer
49 views

Energy function of Restricted Boltzmann Machine (RBM)

The energy function for RBM (Restricted Boltzmann Machine) is defined as $$ E(v,h) = -\sum_{i,j} w_{ij} \, v_i \, h_j -\sum_i a_i \, v_i - \sum_i b_i \, h_i $$ with the joint distribution $$ \tag{1} p(...
2
votes
0answers
19 views

Model or State Uncertainty in Queueing Model due to uncertain arrival rate

$\textbf{Introduction}$ I am currently modelling a scenario where two queues need to be served by a single server in a non preemptive discipline. I am quite sorted on generating the optimal policy ...
2
votes
1answer
62 views

How can we apply the rule of stationary distribution to the continuous case of Markov chain?

If the Markov chain converged then $$\pi = Q* \pi$$where $ \pi$ is the posterior distribution and $Q$ is the transition distribution(it's a matrix in the discrete case). I tried to apply that on the ...
2
votes
1answer
44 views

Hidden-Markov Model for Markov-Chain with Sequential Bernoulli State Sampling

Consider a finite discrete-time Markov chain whose state is sampled at the times determined by the outcome of a Bernoulli process. That is, in each time period I flip a biased coin. If it comes up as "...
3
votes
1answer
42 views

Can we calculate the theoretical stationary distribution from a continuous Markov chain?

I have the transition distribution $p(X_{t+1}|X_t=x_t) = \text{N}(\phi x_t,1)$ where $−1<\phi<1$. Can we calculate the stationary distribution and its mean and variance? I know I can do that ...
0
votes
0answers
85 views

Variance reduction of an estimator arising from the marginal destribution of a Metropolis-Hastings chain

Let $(E,\mathcal E,\lambda)$ and $(E',\mathcal E',\lambda')$ be measure spaces $f\in L^2(\lambda)$ $I$ be a finite nonempty set $\varphi_i:E'\to E$ be bijective $(\mathcal E',\mathcal E)$-measurable ...
1
vote
1answer
48 views

Is the transition kernel of a Metropolis-Hastings chain of the form $P(x,A)=\varrho(x)\tilde P(x,A)+(1-\varrho(x))1_A(x)$?

After equation (1) at page 3 of this paper it is claimed that the transition kernel of a Markov chain generated by the Metropolis-Hastings algorithm is of the form $$P(x,A)=\varrho(x)\tilde P(x,A)+(1-\...
1
vote
1answer
24 views

Birth Death process

An office has two employees that process incoming orders. these two are always busy and they process the orders at the rate of 100/day for each person. However they are smokers. On an average they ...
0
votes
2answers
79 views

Suggest a model for this dataset

I have a time series data set (the Old Faithful geyser data available here: http://www.gatsby.ucl.ac.uk/teaching/courses/ml1-2012/geyser.txt). Plotting the eruption duration on the x axis and the ...
1
vote
2answers
53 views

Could a non-probabilistic state be a random variable?

This post gives a definition "of A stochastic process in discrete time" A stochastic process in discrete time n ∈ $N$ = {0, 1, 2, . . .} is a sequence of random variables (rvs) $X_0, X_1, X_2$, . . ...
-1
votes
1answer
40 views

What is the n of Markov chain exactly equal to?

Section 7.2 of the book "transition probability graph" coming from the book "Introduction to Probability, 2nd Edition by Dimitri P. Bertsekas and John N. Tsitsiklis" gives some explanation of the ...
0
votes
1answer
28 views

Is there a name for this type of transition probability diagram which seems not to be a transition probability graph?

This is a "transition probability graph" coming from the book "Introduction to Probability, 2nd Edition by Dimitri P. Bertsekas and John N. Tsitsiklis". That book also gives this figure, which seems ...
0
votes
0answers
15 views

How many Markov chains are there for 2 states in 1, 2 and 3 steps?

wiki uses this example to illustrate Markov chains. The probabilities of weather conditions (modeled as either rainy or sunny), given the weather on the preceding day, can be represented by a ...
0
votes
1answer
94 views

What is the minimum of n to construct a Markov chain?

Wiki gives this definition of a discrete-time Markov chain a sequence of random variables $X_1$, $X_2$, $X_3$, ... with the Markov property, namely that the probability of moving to the next state ...
0
votes
1answer
19 views

Given a transition matrix P for weather conditions (modeled as either rainy or sunny), is $P^n$ the n-Step Transition Probabilities for day n+1?

wiki uses this example to illustrate Markov chains. The probabilities of weather conditions (modeled as either rainy or sunny), given the weather on the preceding day, can be represented by a ...
0
votes
1answer
24 views

When to exponentiate: the mean of the chain or at every step in the chain?

I am interested in when it is best to exponentiate a difference in log-odds Here is a sample problem in the stan language, three groups of forty binary observations, group 1 with hit probability = 0....
0
votes
0answers
59 views

Make sense of plotting a transition matrix

I'm studying statistics and I'm trying to understand markov chain topic. I'm using the package "markovchain" in R to obtain the stationary distribution. From this transition matrix $M$: ...
0
votes
0answers
11 views

Approximate Value / Policy Iteration (Reinforcement Learning)

I am reading Markov Decision Processes in AI : about Approximate Dynamic Programming. Would you like to explain the rationale for introducing the API algorithm, how it compares to AVI ? How would ...
2
votes
0answers
27 views

Is there a reason why we should run the Metorpolis-Hastings algorithm with a target density approximating the density we're actually after?

Let $(E,\mathcal E,\lambda)$ be a measure space, $p:E\to[0,\infty)$ be $\mathcal E$-measurable with $$c:=\int p\:{\rm d}\lambda$$ and $$\mu:=\underbrace{\frac1cp}_{=:\:\tilde p}\lambda$$ denote the ...
1
vote
0answers
13 views

Can the interarrival times of a continuous time markov chain be distributed with 2 parameter (scale,location) exponential distributions?

I'm trying to model data with a time-homogenous CTMC with a number of states with corresponding constant transition rates $\lambda_{i}$ when I notice that much of the transition times from one state ...
0
votes
0answers
12 views

Markov chain transition Matrix for Ion channels in Python

I was working to code for a markov chain transition matrix for potassium channels. Potassium channels conists of 4 gates 1,2,3,4. the potassium ions antransition from 1->2 or 1->1, 2->1,2->2,2->3 etc. ...
0
votes
1answer
21 views

HMM rolling estimation different from batch estimation

I'm using the GuassianHMM from the python package hmmlearn and after fitting the hmm to the data the predictions that are done in one batch ...
1
vote
0answers
50 views

Sampling with fixed probability from two different distributions. How is the sample distributed?

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $\mu$ be a probability measure on $(\mathbb R,\mathcal B(\mathbb R))$ $X$ be real-valued random variable on $(\Omega,\mathcal A,\...
1
vote
0answers
31 views

How to calculate a 𝑛-step transitions of a Discrete-time Markov Chain for Figure 17.1 (b) in book “Machine Learning - A Probabilistic Perspective”

chapter 17 of the book "Machine Learning - A Probabilistic Perspective" gives this figure which is the probability of getting from i to j in exactly n steps. Obviously A(1) = A. In the case of ...
0
votes
0answers
37 views

What is the point of doing simulation on Markov Chain?

I am studying Markov Chain and I am currently reading about simulation on Markov Chain but I can't see the point of simulation on Markov Chain. What does simulation mean in Markov Chain and what can ...
1
vote
1answer
27 views

can the prior probability change with time?

I was reading Markov Models for sequence modeling and stuck with my understanding(hypothesis). In Baye's theorem, can the prior probabilities change with time? If the prior probability at t=3 is 0.05,...
0
votes
0answers
59 views

Measure the distance between two probability transition matrices

I have a probability transition matrix $P$ that contains some values very close to zero. I want to sparsify this matrix by taking the k largest values for each row and setting the others to zero. For ...
0
votes
0answers
12 views

Markov Chain Monte Carlo for Decryption Purposes

I've been reading the "Markov Chain Monte Carlov Revolution" paper by Diaconis and was intrigued by his application to decryption. I've done some further reading and am now trying to work through ...
1
vote
1answer
17 views

What are the necessary qualifications or assumptions to say that a graph structure is a Markov Chain?

I have a graph structure and want to say it is a Markov Chain. But I am wondering what necessary assumptions or properties that my graph structure need to meet to be called a Markov Chain?
0
votes
0answers
6 views

What are the relationships among Markov Property, Stationarity, and Time Invariance

I am wondering if there is or are any relationship among those. I have understood Markov Property by reading Wikipedia, but it is still confusing to figure out if there is any relationship among those ...