Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

Filter by
Sorted by
Tagged with
0
votes
0answers
44 views
+50

Variance reduction of an estimator arising from the marginal destribution of a Metropolis-Hastings chain

Let $(E,\mathcal E,\lambda)$ and $(E',\mathcal E',\lambda')$ be measure spaces $f\in L^2(\lambda)$ $I$ be a finite nonempty set $\varphi_i:E'\to E$ be bijective $(\mathcal E',\mathcal E)$-measurable ...
1
vote
0answers
11 views

How can we apply the rule of stationary distribution to the continuous case of Markov chain?

If the Markov chain converged then $$\pi = Q* \pi$$where $ \pi$ is the posterior distribution and $Q$ is the transition distribution(it's a matrix in the discrete case). I tried to apply that on the ...
3
votes
1answer
31 views

Can we calculate the theoretical stationary distribution from a continuous Markov chain?

I have the transition distribution $p(X_{t+1}|X_t=x_t) = \text{N}(\phi x_t,1)$ where $−1<\phi<1$. Can we calculate the stationary distribution and its mean and variance? I know I can do that ...
2
votes
1answer
22 views

Hidden-Markov Model for Markov-Chain with Sequential Bernoulli State Sampling

Consider a finite discrete-time Markov chain whose state is sampled at the times determined by the outcome of a Bernoulli process. That is, in each time period I flip a biased coin. If it comes up as "...
1
vote
1answer
23 views

Is the transition kernel of a Metropolis-Hastings chain of the form $P(x,A)=\varrho(x)\tilde P(x,A)+(1-\varrho(x))1_A(x)$?

After equation (1) at page 3 of this paper it is claimed that the transition kernel of a Markov chain generated by the Metropolis-Hastings algorithm is of the form $$P(x,A)=\varrho(x)\tilde P(x,A)+(1-\...
2
votes
1answer
487 views

Planning VS Reinforcement Learning for Large State Spaces

Does knowing everything about your environment yield any major shortcuts to finding the optimal policy, in a Markov Decision Process with a very large (finite) number of states? Mere planning clearly ...
2
votes
2answers
80 views

For a Markov Chain is $𝑃(𝑞_𝑡|𝑞_{𝑡+1},…,𝑞_𝑇)$ equals to $𝑃(𝑞_𝑡| 𝑞_{𝑡+1})$?

I am new to Markov Chains and using this concept in statistics. For a Markov Chain, may I say that $𝑃(𝑞_𝑡|𝑞_{𝑡+1},…,𝑞_𝑇)$ equals to $𝑃(𝑞_𝑡| 𝑞_{𝑡+1})$? If yes, how can I prove that?
0
votes
0answers
13 views

Birth Death process

An office has two employees that process incoming orders. these two are always busy and they process the orders at the rate of 100/day for each person. However they are smokers. On an average they ...
1
vote
1answer
886 views

Measure the arrival rate or inter arrival time for a queueing model

I am analyzing the occurrence of emergencies in a given area and the application of queueing theory to determine the resources an emergency service should have ready in order to answer to emergency ...
0
votes
2answers
76 views

Suggest a model for this dataset

I have a time series data set (the Old Faithful geyser data available here: http://www.gatsby.ucl.ac.uk/teaching/courses/ml1-2012/geyser.txt). Plotting the eruption duration on the x axis and the ...
1
vote
2answers
50 views

Could a non-probabilistic state be a random variable?

This post gives a definition "of A stochastic process in discrete time" A stochastic process in discrete time n ∈ $N$ = {0, 1, 2, . . .} is a sequence of random variables (rvs) $X_0, X_1, X_2$, . . ...
2
votes
1answer
342 views

How is the Fermiac machine (Monte Carlo trolley) working?

There is a cool website showing the Markov chain with a machine. But nobody is explaining how it's working or showing a video of it's functioning. This is explaining the Markov chain monte carlo in ...
-1
votes
1answer
40 views

What is the n of Markov chain exactly equal to?

Section 7.2 of the book "transition probability graph" coming from the book "Introduction to Probability, 2nd Edition by Dimitri P. Bertsekas and John N. Tsitsiklis" gives some explanation of the ...
3
votes
0answers
554 views

How to estimate Markov chain transition probabilities with partially observed data?

Suppose that we have a time-homogeneous discrete-time Markov chain $(X_n)$. We want to estimate the transition probabilities $p_{ij} = \mathbb{P}[X_{n+1} = j \mid X_n = i]$. In the case when we have ...
1
vote
1answer
149 views

r markov chain markov property on binary variable, discrete time

i have a sequence of 1/0's indicating if patient is in remission or not, assume the records of remission or not were taken at discrete times, how can i check the markov property for each patient, ...
0
votes
1answer
79 views

What is the minimum of n to construct a Markov chain?

Wiki gives this definition of a discrete-time Markov chain a sequence of random variables $X_1$, $X_2$, $X_3$, ... with the Markov property, namely that the probability of moving to the next state ...
0
votes
1answer
27 views

Is there a name for this type of transition probability diagram which seems not to be a transition probability graph?

This is a "transition probability graph" coming from the book "Introduction to Probability, 2nd Edition by Dimitri P. Bertsekas and John N. Tsitsiklis". That book also gives this figure, which seems ...
0
votes
1answer
16 views

Given a transition matrix P for weather conditions (modeled as either rainy or sunny), is $P^n$ the n-Step Transition Probabilities for day n+1?

wiki uses this example to illustrate Markov chains. The probabilities of weather conditions (modeled as either rainy or sunny), given the weather on the preceding day, can be represented by a ...
0
votes
0answers
14 views

How many Markov chains are there for 2 states in 1, 2 and 3 steps?

wiki uses this example to illustrate Markov chains. The probabilities of weather conditions (modeled as either rainy or sunny), given the weather on the preceding day, can be represented by a ...
0
votes
1answer
19 views

When to exponentiate: the mean of the chain or at every step in the chain?

I am interested in when it is best to exponentiate a difference in log-odds Here is a sample problem in the stan language, three groups of forty binary observations, group 1 with hit probability = 0....
3
votes
1answer
1k views

From Markov Decision Process (MDP) to Semi-MDP: What is it in a nutshell?

Markov Decision Process (MDP) is a mathematical formulation of decision making. An agent is the decision maker. In the reinforcement learning framework, he is the learner or the decision maker. We ...
16
votes
3answers
31k views

Estimating Markov transition probabilities from sequence data

I have a full set of sequences (432 observations to be precise) of 4 states $A-D$: eg $$Y=\left(\begin{array}{c c c c c c c} A& C& D&D & B & A &C\\ B& A& A&C &...
0
votes
0answers
49 views

Make sense of plotting a transition matrix

I'm studying statistics and I'm trying to understand markov chain topic. I'm using the package "markovchain" in R to obtain the stationary distribution. From this transition matrix $M$: ...
0
votes
0answers
10 views

Approximate Value / Policy Iteration (Reinforcement Learning)

I am reading Markov Decision Processes in AI : about Approximate Dynamic Programming. Would you like to explain the rationale for introducing the API algorithm, how it compares to AVI ? How would ...
1
vote
0answers
23 views

Is there a reason why we should run the Metorpolis-Hastings algorithm with a target density approximating the density we're actually after?

Let $(E,\mathcal E,\lambda)$ be a measure space, $p:E\to[0,\infty)$ be $\mathcal E$-measurable with $$c:=\int p\:{\rm d}\lambda$$ and $$\mu:=\underbrace{\frac1cp}_{=:\:\tilde p}\lambda$$ denote the ...
0
votes
0answers
10 views

Can the interarrival times of a continuous time markov chain be distributed with 2 parameter (scale,location) exponential distributions?

I'm trying to model data with a time-homogenous CTMC with a number of states with corresponding constant transition rates $\lambda_{i}$ when I notice that much of the transition times from one state ...
0
votes
0answers
6 views

Markov chain transition Matrix for Ion channels in Python

I was working to code for a markov chain transition matrix for potassium channels. Potassium channels conists of 4 gates 1,2,3,4. the potassium ions antransition from 1->2 or 1->1, 2->1,2->2,2->3 etc. ...
0
votes
1answer
15 views

HMM rolling estimation different from batch estimation

I'm using the GuassianHMM from the python package hmmlearn and after fitting the hmm to the data the predictions that are done in one batch ...
0
votes
0answers
45 views

Sampling with fixed probability from two different distributions. How is the sample distributed?

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $\mu$ be a probability measure on $(\mathbb R,\mathcal B(\mathbb R))$ $X$ be real-valued random variable on $(\Omega,\mathcal A,\...
1
vote
0answers
31 views

How to calculate a 𝑛-step transitions of a Discrete-time Markov Chain for Figure 17.1 (b) in book “Machine Learning - A Probabilistic Perspective”

chapter 17 of the book "Machine Learning - A Probabilistic Perspective" gives this figure which is the probability of getting from i to j in exactly n steps. Obviously A(1) = A. In the case of ...
0
votes
0answers
32 views

What is the point of doing simulation on Markov Chain?

I am studying Markov Chain and I am currently reading about simulation on Markov Chain but I can't see the point of simulation on Markov Chain. What does simulation mean in Markov Chain and what can ...
1
vote
1answer
27 views

can the prior probability change with time?

I was reading Markov Models for sequence modeling and stuck with my understanding(hypothesis). In Baye's theorem, can the prior probabilities change with time? If the prior probability at t=3 is 0.05,...
0
votes
0answers
33 views

Measure the distance between two probability transition matrices

I have a probability transition matrix $P$ that contains some values very close to zero. I want to sparsify this matrix by taking the k largest values for each row and setting the others to zero. For ...
1
vote
0answers
62 views

Does the Markov property always hold for a state-space structure?

Markov Property: $p({\bf x}_t | {\bf x}_1, \ldots, {\bf x}_{t-1}) = p({\bf x}_t | {\bf x}_{t-1})$ Consider the following model for which the hidden states are ${\bf x}_t$ and the observations are ${\...
0
votes
0answers
12 views

Markov Chain Monte Carlo for Decryption Purposes

I've been reading the "Markov Chain Monte Carlov Revolution" paper by Diaconis and was intrigued by his application to decryption. I've done some further reading and am now trying to work through ...
1
vote
1answer
16 views

What are the necessary qualifications or assumptions to say that a graph structure is a Markov Chain?

I have a graph structure and want to say it is a Markov Chain. But I am wondering what necessary assumptions or properties that my graph structure need to meet to be called a Markov Chain?
0
votes
0answers
6 views

What are the relationships among Markov Property, Stationarity, and Time Invariance

I am wondering if there is or are any relationship among those. I have understood Markov Property by reading Wikipedia, but it is still confusing to figure out if there is any relationship among those ...
0
votes
0answers
24 views

How to train a single discrete-time markov model?

I have a training set of sequences. I want to reach a discrete time Markov model (transition probability matrix). Is there a Bayesian way other than MLE to achieve this?
2
votes
2answers
59 views

Proof of (weak) consistency for an unbiased estimator

I want to prove a theorem stating: An unbiased estimator $\hat{\theta}$ of the unknown parameter $\theta$ is consistent if $V(\hat{\theta}_n$) $\to0$ for ${n\to\infty}$. I've tried using the ...
2
votes
1answer
1k views

Predicting the next value

I've recently read a blogpost where someone tries to predict the outcome of eurovision. Quoting from the summary of the site: Essentially, we can look at people’s voting preferences in the ...
3
votes
1answer
53 views

Spread of number of steps to reach absorbing state in markov chain

I know how to calculate the variance of the number of steps in an absorbing markov chain. However, I am not sure that the distribution of the number of steps is normal. Therefore I would like to ...
0
votes
1answer
209 views

Issue in graph construction

I have a symbolic representation of time series obtained from SAX toolbox. I was wondering if it is possible to construct a graph where each node represents a unique symbol and the edges represent the ...
1
vote
1answer
210 views

Question about marked poisson process

Let's say I have a Poisson point process on $\left[0,T\right]$ with rate $\lambda\left(t\right)=2t^2$. Suppose I attach a mark $m_t$ to each point $t$ of the process such that $m_t\sim N\left(t,1\...
1
vote
1answer
626 views

Markov Chain and Removal Effect

Based on this article I'm trying to use within R the Channel Attribution package to leverage on the Markov Chain in order to attribute conversion between several marketing channel. On one point the ...
0
votes
0answers
19 views

How to find exact month of regime shifts in Markov switching model in R?

I'm looking into industry specific merger waves. I have used the Markov switching model (AR(0)) to identify periods with high M&A activity (merger wave periods) and periods with lower/normal M&...
2
votes
1answer
329 views

Measuring dependency of subsequent points from Markov chain

The question is about stimulating different type of species (coded 1-10) based on given species frequencies, and other parameters (eg. mean of normally distributed mass and ratio) using gibbs sampling....
4
votes
1answer
273 views

Can a Markov chain be approximated with an AR process?

In some MCMC literature/source code, a Markov chain is often approximated with an AR(1) process. There is some theory to suggest that such an approximation is somewhat valid for a finite state space, ...
2
votes
0answers
196 views

A Hidden Markov model with covariates in the transition probabilities

I would like to construct a Hidden Markov model with data about online customer journeys. A well-known concept related to the customer journey literature is the sales funnel. Consumers walk through ...